How do you track down the source of high CPU usage?

[PancakeBimmer]

Over the holidays something has happened with BlueIris computer and CPU usage sky-rocketed.
After couple hours of looking at resource usage the only thing I found was memory usage close to 8GB.
At the same time frame rate on all cameras dropped to 1-3 with immense lag. Note that at this time there were no local or remote sessions on the computer, only UI3 open. Driver version is 24.20.100.6229 so it can't be memory leak since this version is one of the recommended ones.

After some time the CPU usage settled down to normal levels of ~50% and I still have no clue what was causing the usage. How do I go about this next time?

Setup consists of 14 4-5MP cameras with direct to disk recording at 15FPS (couple at 20-25).
Version: Release 4.8.4.5 x64
OS: Windows 10 Pro
CPU: Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz [42%]

 

[mat200]

Hi @PancakeBimmer

Check your system's vitals - often if the RAM gets completely used, something gets swapped out of RAM and goes to HDD.. which really hurts performance.
( thus another reason why SSD is nice for the system disk )

 

[PancakeBimmer]

Hi @PancakeBimmer

Check your system's vitals - often if the RAM gets completely used, something gets swapped out of RAM and goes to HDD.. which really hurts performance.
( thus another reason why SSD is nice for the system disk )

System has 16GB of ram with blueiris using 8GB and during the issues system overall used 70% of ram.
I was suspecting that blueiris might be using hard drive as a swap space, but I don't exactly know how to check if that is the case.
System is installed on NVME SSD.

Edit: My mistake, I forgot that most of cameras are 8MP and not what I previously stated

 

[SouthernYankee]

Start with a reboot.

Do you have at least 20% free space on the system drive?

Is this a new problem or did it just start.

Read the CPU secition in the wiki.

Screen shot of the tabs in the windows task manager.

 

[PancakeBimmer]

Start with a reboot.

Do you have at least 20% free space on the system drive?

Is this a new problem or did it just start.

Read the CPU secition in the wiki.

Screen shot of the tabs in the windows task manager.

Restart is first thing I tried and unfortunately it did not resolve anything in this case.
System drive has 75% of storage free.
Completely new problem just started randomly without any changes to the system or settings.
Went though that CPU optimization section just in case I missed anything, and all except the "limit decoding" optimizations are applied.

This morning usage is usual around 60% with no issue and smooth feed, and this is with me remotely connected with unrestricted remote fps.

 

[SouthernYankee]

It looks like you may be overloading the GPU.

Go slow, test do not guess.... do each of the following one at a time for at least a day. to see if it improves anything. Look at the GPU load.

1) On cameras that are recording continuous direct to disk, with out any addition processing (no BI motion detection...) check the "Limit decoding unless required" on the camera video tab.
2) on a few cameras that are not critical, no additional processing(no bi motion detection...) turn off the hardware acceleration.


Are you using the camera time stamp on the video or the BI time stamp ? I turned off the BI time stamp and reduced my cpu load (enable overlays uncheck).

 

[bp2008]

This looks to me like you are just running up against the limit of what that CPU can handle. A while back someone tested the limits of an i7-8700K and found it couldn't really handle more than 1500 MP/s. Exceeding that tended to break it much like you have seen with your load (showing 1570 MP/s in yesterday's screenshot). If you took 2 or 3 of the 8 MP cameras down to 10 FPS so as to stay under 1500 MP/s total, that should make it more stable. Taking away 5 FPS from one 8 MP camera should reduce the total by 40 MP/s.

Your GPU video decode utilization is nearly 100%, so it is possible it is not always able to keep up with all the incoming video. When that happens, frames start getting buffered longer. Video delay goes up. Memory usage goes up. Blue Iris may try to compensate by dropping frames (I really don't know if it is coded to do that or not) but doing so would introduce video corruption and make the frame rate appear to have dropped drastically on one or more cameras.

Now, disabling hardware acceleration on a few cameras is an option, but despite what it looks like, you really don't have much CPU time left over. The i7-8700K has hyperthreading which means the CPU pretends to have 12 cores when it really only has 6. You can think of it this way. From 0 to 50% you are using the real cores, and you get a fairly predictable amount of work done at any % utilization up to 50%. But after 50% you start dipping into the "vitual cores" provided by hyper-threading. These "virtual cores" are only the result of some clever scheduling tricks, and are only capable of a fraction of a real core's workload. But Windows treats the virtual cores as equal to the real ones. The result is that as you add work to the CPU, its utilization % rises much faster and less predictably once you pass 50%. So if you are at or above 50% CPU already on a CPU with hyper-threading, then you are almost to the limit already.

 

[PancakeBimmer]

This looks to me like you are just running up against the limit of what that CPU can handle. A while back someone tested the limits of an i7-8700K and found it couldn't really handle more than 1500 MP/s. Exceeding that tended to break it much like you have seen with your load (showing 1570 MP/s in yesterday's screenshot). If you took 2 or 3 of the 8 MP cameras down to 10 FPS so as to stay under 1500 MP/s total, that should make it more stable. Taking away 5 FPS from one 8 MP camera should reduce the total by 40 MP/s.

Your GPU video decode utilization is nearly 100%, so it is possible it is not always able to keep up with all the incoming video. When that happens, frames start getting buffered longer. Video delay goes up. Memory usage goes up. Blue Iris may try to compensate by dropping frames (I really don't know if it is coded to do that or not) but doing so would introduce video corruption and make the frame rate appear to have dropped drastically on one or more cameras.

Now, disabling hardware acceleration on a few cameras is an option, but despite what it looks like, you really don't have much CPU time left over. The i7-8700K has hyperthreading which means the CPU pretends to have 12 cores when it really only has 6. You can think of it this way. From 0 to 50% you are using the real cores, and you get a fairly predictable amount of work done at any % utilization up to 50%. But after 50% you start dipping into the "vitual cores" provided by hyper-threading. These "virtual cores" are only the result of some clever scheduling tricks, and are only capable of a fraction of a real core's workload. But Windows treats the virtual cores as equal to the real ones. The result is that as you add work to the CPU, its utilization % rises much faster and less predictably once you pass 50%. So if you are at or above 50% CPU already on a CPU with hyper-threading, then you are almost to the limit already.

Damn, you are right. Dropping the frame rate to 10 on all cameras lowered the MP/s to 1100 and system is stable again. I don't suppose there are any remedies to this other than trying different software solution or splitting up cameras into multiple systems? Boss really likes the details from 8MP cameras, but as we learned it comes with the added cost.

 

[bp2008]

There are plenty of remedies. First of all you likely don't need to reduce frame rates on so many cameras to make it stable. Some of your cameras are probably less important. Let the less important cameras take the hit.

The Limit Decode option makes a huge difference if you can get away with having slow and delayed live views for certain cameras. Enabling this does strongly impact the effectiveness of BI's motion detection, but it does not impair the frame rate of the recordings in any way. Limit Decode must be used carefully in order to not backfire when you use the mobile apps or web interface (UI3). If you use Limit Decode on any cameras, you should disable the "all frames" requirement for all your camera groups, here:

Or you could build a faster system. i7-9900K is the most efficient upgrade, as that is just like your current CPU but with two more cores. It should take care of the current problem, but not give you a huge amount of headroom for future expansion, and the upgrade would be costly even if you could re-use the same motherboard and cooler. A more powerful and more expensive choice would be a high-end AMD threadripper CPU or a high-end Intel HEDT CPU (like those with 18 cores for $2000). With those CPUs you lose Intel Quick Sync hardware acceleration but gain a lot more raw processing power, and if you really have a huge budget you can add Nvidia graphics cards to offload the decoding work to further expand the capabilities of a single system. At that point you'd be spending thousands of dollars and consuming 200-1000 watts continuously though. In any case, a faster system costs more so it isn't a good option for most, especially if you need the very fastest CPUs which don't support Quick Sync.

 

[PancakeBimmer]

There are plenty of remedies. First of all you likely don't need to reduce frame rates on so many cameras to make it stable. Some of your cameras are probably less important. Let the less important cameras take the hit.

The Limit Decode option makes a huge difference if you can get away with having slow and delayed live views for certain cameras. Enabling this does strongly impact the effectiveness of BI's motion detection, but it does not impair the frame rate of the recordings in any way. Limit Decode must be used carefully in order to not backfire when you use the mobile apps or web interface (UI3). If you use Limit Decode on any cameras, you should disable the "all frames" requirement for all your camera groups, here:

Or you could build a faster system. i7-9900K is the most efficient upgrade, as that is just like your current CPU but with two more cores. It should take care of the current problem, but not give you a huge amount of headroom for future expansion, and the upgrade would be costly even if you could re-use the same motherboard and cooler. A more powerful and more expensive choice would be a high-end AMD threadripper CPU or a high-end Intel HEDT CPU (like those with 18 cores for $2000). With those CPUs you lose Intel Quick Sync hardware acceleration but gain a lot more raw processing power, and if you really have a huge budget you can add Nvidia graphics cards to offload the decoding work to further expand the capabilities of a single system. At that point you'd be spending thousands of dollars and consuming 200-1000 watts continuously though. In any case, a faster system costs more so it isn't a good option for most, especially if you need the very fastest CPUs which don't support Quick Sync.

We have a similarly spec system becoming available in couple days (old location closed down), so at this point plan is to split the load between two systems considering that are will be 6-8 more 8MP cameras added.
On a side note, does anyone have knowledge why this overload happened occasionally? Overnight with very little motion it was running smoothly, and as people started coming in the morning the usage went up and up until the issues described in original post. However, after 3-4 hours it smoothed out again until the next morning.
I would guess that more motion might result in more decoding work to be done hence they lag, however as already pointed out there was still same amount of motion after 3-4 hours when system starts running smooth again.

How crazy would it be if 8700k or 9900K could be dual-socket and both GPUs would be available to blue iris.