Stuttering video

[Tayschrenn]

I'm at a loss on this one. Video on my BI is super laggy when motion occurs, to the point that you can see time skip/catch up - but the box isn't overloaded at all.

An example link (google photos - tried uploading directly but it wouldn't complete)

Hardware:
i5-9500
8Gb RAM
Cache drive is nvme (initial writes go to this)
2nd stage is a 7200rpm Enterprise SATA (after nvme fills it moves it)
Using substream to minimize overhead


Camera's are IPC-HFW2831T-ZS 8MP
Camera settings:



BI Camera settings:





Using QSV and it's plenty powerful of a CPU for 3x4k cameras.

Connectivity is via Unifi switches, using cat 6 shielded cabling so i'm confident it's not connectivity.

 

[wittaj]

Match iframe rate to FPS and make the substream code match mainstream - H264.

And I think your bitrate is too low - up it to 8192 at a minumum. If that doesn't fix it, then go to the maximum.

If that doesn't fix it, then look at your network.

Are the cams going through the router to get to the BI computer? Routers are not efficient at routing these types of cameras.

If your cameras go thru the router to get to the BI computer, the router is likely being overloaded as they cannot keep up with the constant stream of video. These cameras are not like streaming services - these cameras do not buffer like NetFlix does....pull your internet and the NetFlix will still stream for up to 30 seconds...pull the internet on a camera and you lose the stream IMMEDIATELY...

 

[Tayschrenn]

Are the cams going through the router to get to the BI computer? Routers are not efficient at routing these types of cameras.

If your cameras go thru the router to get to the BI computer, the router is likely being overloaded as they cannot keep up with the constant stream of video. These cameras are not like streaming services - these cameras do not buffer like NetFlix does....pull your internet and the NetFlix will still stream for up to 30 seconds...pull the internet on a camera and you lose the stream IMMEDIATELY...

I'll adjust the bitrates and FPS/etc - thanks - is 512Kb/S fine for the sub stream?

Do I match BI FPS or leave it at 30?

It's not going through a router, it's all on one layer-2 vlan and all hard wired. While it's technically a routed VLAN, the stream isn't leaving the layer-2 and the BI computer is connected to the switch the cameras are on via a LACP 2gbps fiber connection, so there's no chance of latency between switches either. The computer currently only runs a single Intel L219-LM adapter but it doesn't use more than 40-60Mbps anyways.

If that becomes a concern, I upgraded to an Optiplex 7070 specifically so I can use the PCI slots for things like a quad port adapter, and/or add a GPU in the future to offload file exports etc (though not until BI is better suited to use nvec).

Well and because I picked it up as B-Stock off CDW outlet for $315 haha. That was a lucky find tbh, given how much faster the i5-9500 is compared to my old i3-7100T.

 

[TonyR]

Cache drive is nvme (initial writes go to this)
2nd stage is a 7200rpm Enterprise SATA (after nvme fills it moves it)

On which drive is the O/S, the BI program and the BI db folder installed?

 

[wittaj]

I'll adjust the bitrates and FPS/etc - thanks - is 512Kb/S fine for the sub stream?

Do I match BI FPS or leave it at 30?

It's not going through a router, it's all on one layer-2 vlan and all hard wired. While it's technically a routed VLAN, the stream isn't leaving the layer-2 and the BI computer is connected to the switch the cameras are on via a LACP 2gbps fiber connection, so there's no chance of latency between switches either. The computer currently only runs a single Intel L219-LM adapter but it doesn't use more than 40-60Mbps anyways.

If that becomes a concern, I upgraded to an Optiplex 7070 specifically so I can use the PCI slots for things like a quad port adapter, and/or add a GPU in the future to offload file exports etc (though not until BI is better suited to use nvec).

Well and because I picked it up as B-Stock off CDW outlet for $315 haha. That was a lucky find tbh, given how much faster the i5-9500 is compared to my old i3-7100T.

Yeah, the substream comes down to how many cameras you have on the screen, I run 256 for all of mine with no issue.

Leave the FPS in BI alone - it will adjust that automatically.

I think matching FPS and iframe in the camera (try 15 for each) and a higher bitrate will take care of the issue.

 

[Tayschrenn]

On which drive is the O/S, the BI program and the BI db folder installed?

OS/BI/DB is all on nvme.

Yeah, the substream comes down to how many cameras you have on the screen, I run 256 for all of mine with no issue.

Leave the FPS in BI alone - it will adjust that automatically.

I think matching FPS and iframe in the camera (try 15 for each) and a higher bitrate will take care of the issue.

Doesn't look like it has fixed it, still super laggy on that camera - but oddly enough NOT on the identical camera watching from the other corner of the house. Same exact settings on both, except that the non laggy camera has a different trigger / recording profile:

So this one has hour long recordings, but I don't know why that should matter.

View attachment SE.20210505_190000_1.mp4

View attachment NE.20210505_195103_1.mp4

 

[wittaj]

This shouldn't do it, but triggered+periodic each probably isn't doing what you think it is. Try either just triggered or continuous.

How are the cams powered - I have seen this also happen when one camera is underpowered? Or a bad cable.

Open up the camera status in BI and watch the FPS and KEY and see if they hold steady at your FPS. Sometimes we can see the FPS fluctuate due to power issues.

If they are all on the same switch, I would unplug all the cameras but the problem one and see if that fixes it and then you know it is a power issue.

And maybe try with the overlays off. What is the CPU % during motion?

And kind of a pain, but you can always take the cam down and test it with a known good cable.

 

[Tayschrenn]

This shouldn't do it, but triggered+periodic each probably isn't doing what you think it is. Try either just triggered or continuous.

How are the cams powered - I have seen this also happen when one camera is underpowered? Or a bad cable.

Open up the camera status in BI and watch the FPS and KEY and see if they hold steady at your FPS. Sometimes we can see the FPS fluctuate due to power issues.

If they are all on the same switch, I would unplug all the cameras but the problem one and see if that fixes it and then you know it is a power issue.

And maybe try with the overlays off. What is the CPU % during motion?

And kind of a pain, but you can always take the cam down and test it with a known good cable.

I see you're one of those people who hate life.


It actually looks like it's using MORE power than the other camera. I don't show any rx/tx errors on the port either.
Edit: And the switch is fine - this unit can support 150w PoE and i'm only using 29w total at the moment. If I need to I can switch it to my other switch (same model) as a variable.



CPU wise it's fine, this rarely goes over 30% even with me doing stuff on the box / console open.

I did just get new firmware from Andy as this was using flash still, they're now on V2.800.0000000.5.R, Build Date: 2020-01-07 - i'll see if this makes any changes.

I'm not sure what you mean by FPS and KEY - where is that at? In general though things look fine, I don't have crazy ping latency or anything (or at least it's inline with pinging other LAN items so generally <1ms with 3-4MS high)

It's 'possible' the cable is having issues, I didn't test this one but I can borrow a cable tester from the office once it warms up some and test the cable, if the issues don't resolve.

 

[wittaj]

Since you have a Dahua cam, we know that we can actually set the parameters, unlike some other to remain nameless cams LOL.

From the BI status screen for the cameras, watch the FPS/key and they should basically hover around what you put in for FPS. The KEY is a ratio, so if the KEY is 1.00 that means your FPS and iframes match. If KEY is 0.5 that means your Iframes is double the FPS. I had a camera that would bounce around 4FPS and that is a clear sign of a power issue with the Dahua...although if it doesn't bounce around doesn't mean that it still isn't a power issue.



That is interesting that the same cam is using more power.

 

[Tayschrenn]

Since you have a Dahua cam, we know that we can actually set the parameters, unlike some other to remain nameless cams LOL.

From the BI status screen for the cameras, watch the FPS/key and they should basically hover around what you put in for FPS. The KEY is a ratio, so if the KEY is 1.00 that means your FPS and iframes match. If KEY is 0.5 that means your Iframes is double the FPS. I had a camera that would bounce around 4FPS and that is a clear sign of a power issue with the Dahua...although if it doesn't bounce around doesn't mean that it still isn't a power issue.

View attachment 88911

That is interesting that the same cam is using more power.

You nerds are awesome.


Didn't know this existed until just now. I have it set to 20 (max of 30 on these cameras) and it looks to be good - the one having issues is NE/NorthEast.

I'll have my wife go jump around in the morning so i can watch the box and status while motion is detected. Thanks a ton for all the great information.

p.s. I figured out the PoE - firmware update apparently re-enabled Auto on the IR for that camera, so it was using more power to run the higher IR setting. Fixed that now they're both around 5.

 

[wittaj]

Oh - the NE cam is using Hardware Acceleration (HA) and the other ones are not - turn that off in the camera setting in BI and I bet the problem goes away. Some people have been experiencing issues with HA on recent versions.

 

[Tayschrenn]

Oh - the NE cam is using Hardware Acceleration (HA) and the other ones are not - turn that off in the camera setting in BI and I bet the problem goes away. Some people have been experiencing issues with HA on recent versions.

That seems... backwards - isn't Intel VPP ideal for smoothing things out and helping with CPU utilization?

I'll do that next though if the current changes do not resolve - don't want to change too many variables at once.

 

[fenderman]

That seems... backwards - isn't Intel VPP ideal for smoothing things out and helping with CPU utilization?

I'll do that next though if the current changes do not resolve - don't want to change too many variables at once.

if you overload the decoder it cam create bottleneck issues. This is particularly true with intel-vpp. Try plain intel.

 

[wittaj]

I have tried many things over the years, and I personally haven't noticed a change in video quality in terms of smoothing things out whether GPU is used or not. Some of my cams don't like it. But as mentioned, it can be overloaded, and some people have experienced issues with newer updates of BI and HA on some cameras - could be a driver issue or incompatibility issue and purely coincidental with the update.

But your other cams are not using HA and your problem one is.

And may or may not be an issue, but look over to the far right of your cam status and how many No Signals your cams are showing - a rate of one no signal every minute. Could be absolutely nothing, but it could also be a power issue or computer issue and that camera just happens to be more sensitive to the power fluctuations.

 

[Tayschrenn]

I have tried many things over the years, and I personally haven't noticed a change in video quality in terms of smoothing things out whether GPU is used or not. Some of my cams don't like it. But as mentioned, it can be overloaded, and some people have experienced issues with newer updates of BI and HA on some cameras - could be a driver issue or incompatibility issue and purely coincidental with the update.

But your other cams are not using HA and your problem one is.

And may or may not be an issue, but look over to the far right of your cam status and how many No Signals your cams are showing - a rate of one no signal every minute. Could be absolutely nothing, but it could also be a power issue or computer issue and that camera just happens to be more sensitive to the power fluctuations.

I set them all to Intel standard and will try it out, then disable it and try.

Does it use QSV natively? I'd assumed that was the hardware offload.

While this isn't a xeon with the p630 iGPU, or the delicious new Xe graphics, the UHD 630 should still be able to handle 3 cameras I'd have thought.

Thanks again. I'll update as I run tests.

 

[Tayschrenn]

if you overload the decoder it cam create bottleneck issues. This is particularly true with intel-vpp. Try plain intel.

Is this a metric recorded in perf mon? How do you evaluate overload - task manager doesn't ever show much use in the iGPU even with both decoders displayed, except when the console is open (which I rarely have open, at most I'll use the web UI)

 

[wittaj]

Hard headed I see LOL - you have 3 cameras and one is wonky and is found that only that one is using HA, so instead of turning HA off on that one, you turn HA on for all three LOL.

With 3 cameras, using the GPU isn't going to make much of an impact on the CPU, nor show much usage on the GPU.

And what part are you missing that we have said that some have experienced issues using HA with some cameras or some setups? Many are running a computer much older than yours with more cameras and HA is fine, while others have newer computers and cannot run HA on them all - so many variables that can cause it to be problematic for some (but most of the time it is user error LOL).

And why are your cameras losing signal at a rate of once per minute?

 

[The Automation Guy]

The stuttering issues you are seeing are playback issues. If it was stuttering while recording, the car would jump forward because there would be missing footage. It looks like this happens one time in the second video, but primarily you are seeing playback that is stuttering. All of the footage is there, it just isn't being played back in one continuous stream.

 

[Tayschrenn]

Hard headed I see LOL - you have 3 cameras and one is wonky and is found that only that one is using HA, so instead of turning HA off on that one, you turn HA on for all three LOL.

With 3 cameras, using the GPU isn't going to make much of an impact on the CPU, nor show much usage on the GPU.

And what part are you missing that we have said that some have experienced issues using HA with some cameras or some setups? Many are running a computer much older than yours with more cameras and HA is fine, while others have newer computers and cannot run HA on them all - so many variables that can cause it to be problematic for some (but most of the time it is user error LOL).

And why are your cameras losing signal at a rate of once per minute?

Variable logic - if it were HA causing the issue, all cameras would in turn show the issue with it enabled (more so the SE camera as it's identical) - this isn't the case however, with HA on I am still getting the stutter only on the NE camera.

Not losing any packets - constant ping to the camera - and while there's latency it's not over 70ms

As for losing connection - I'm not showing that it's losing connection once per minute - where is that metric located? Must be blind / not understanding the syntax of a screen.

While typing this I actually found that inside BI settings there's a tab called 'camera' that also had Intel + VPP selected for HA there - I've disabled that / set it to 'No' and restarted the cameras.

I'll see if that does anything however, now that I know the other changes I've made (i.e. firmware) have not resolved the issue.

My next step before re-wiring will be to connect the camera on my Synology Surveillance Station and see if it does the same thing there. In theory if it's network, it'll show the same issue in both NVRs.

The stuttering issues you are seeing are playback issues. If it was stuttering while recording, the car would jump forward because there would be missing footage. It looks like this happens one time in the second video, but primarily you are seeing playback that is stuttering. All of the footage is there, it just isn't being played back in one continuous stream.

While I understand playback and conversion can absolutely cause this, I've tried different exports and they all show the same thing. Locally playback from the BVR format also shows the stutter. Unless that's then something that BI is causing when writing to the BVR (i.e. HA? if so why wouldn't all the cameras show the same issue with HA enabled?)

I dislike issues that don't have debug logs i can dig through.

 

[wittaj]

Hard headed LOL - some cameras will be buggy with HA, so saying "if it were HA causing the issue, all cameras would in turn show the issue with it enabled" is an inaccurate statement. It can be one camera (and even one of the same model as it could be a different chipset for example between your two "identical" cameras) that is whacked out using HA with all the remaining cameras using HA just fine.

Many here have that problem camera that just doesn't like HA. Sometimes BI will recognize that and turn HA off for the problem camera, and sometimes we just have to turn it off ourself.

Hopefully you tried all cameras without HA to see if that fixed the problem. Like I said, for 3 cameras and your 9th generation CPU, HA isn't going to make a big impact. GPU usage with HA is probably 1% or so.

No signal - far right column of your Camera Status tab. The problem camera at this point had been online for a whopping 31 minutes and had 30 Loss of signals (but all of your cameras are showing loss of signal)...like I said, it could be nothing, or it is a power or setup issue that BI is taking an exception too and this particular camera is more susceptible to fluctuations than others...just because they are identical cameras doesn't mean much - as you saw the problem camera was using a lot more power than the other - that could be due to the scene differences (although they were both basically looking at the same thing) or maybe a different chipset between the two, or a variety of other issues.

As part of troubleshooting, I would put the BI computer and the cameras on the same switch temporarily and see if the No Signal goes away.




Sometimes simply deleting the camera from BI and adding it back can fix a lot of problems. Probably worth a try as well.