bottleneck

[uknod]

Hi, This may be a question for an IT expert who understands how computers work, rather than an expert in Blue Iris, as it has got even Blue Iris support baffled.
This is the problem:

When increasing the number of web logins and the number of cameras, at a point the system starts to lower the frame rate massively, however there are no bottle necks at all. CPU, GPU, Network, Hard Drive, Memory all less than <50%, yet frame rate can go down from 20fps to about 1 frame per 15 seconds.

This is the hardware:
xeon dual processor with 32cores. Quadro P2000 graphics, 32GB memory

Cameras 16 x 8k cameras (which have to run on substream (1200 x 536 in order to have each one at 20fps.)

16 web connections to view each individual camera.

In the above scenario, if I change just one camera to 8k (main stream), then the frame rate will drastically drop on all cameras. If I switch 2 cameras to 8k then the 8k cameras will practically stop. By the time I have switched 4 cameras to 8k then the whole system stops. However as stated above, non of the resources goes over 50%, so my quest is:

Where is the bottle neck.

 

[TonyR]

Have you checked your network's bandwidth ? (speed / capacity)

Is the hardware 10/100 Megabit?

 

[alastairstevenson]

Also, suggest using resource monitor to look carefully at how the HDDs are working, especially the active time and the latency.

 

[SouthernYankee]

Are the cameras Wifi? Are the login connections wifi ? Provide a network drawing.

 

[Flintstone61]

Just one Neolithic cavemans observation.
One bottleneck I can think of is using 1 hard drive. I'd be using multiple hard drives for that much thruput.
It may not be ALL of your issue, but in my experience my Blue iris systems are more responsive when the hard drives are not bogged down.
I guess i dont understand the " web logins" portion of the problem, but if your viewing 16 cameras on 16 seperate internet streams you'd need Fiber internet or a very fast copper internet with high Up/down bit rates

 

[plomaris]

What version of BI are you using? I had an issue with version 6 where one camera that is viewed remotely 24x7 would freeze up after a few hours. On the Camera settings, General tab, it would say "frame bottleneck". After many troubleshooting steps, I reverted back to v5 and that seemed to solve the problem. BI Support has been notified.

 

[bp2008]

I don't think dual processor systems are supported well by Blue Iris (or most other software for that matter). If CPU load is even close to 50%, that is more like 100%. I don't think Blue Iris can go meaningfully over 50% on a dual CPU system, is what I'm saying.

Also, if it is a dual processor Xeon, that probably means it is 10+ years old (otherwise you likely wouldn't have been able to afford it), and older CPUs aren't very fast no matter how many cores they have. What CPU model is it exactly?

You say "16 web connections to view each individual camera" but what are those connections to? To Blue Iris's web server? You have 16 connections to each of 16 cameras (16x16 = 256)??? Can you clarify this? Because even 16 connections is more than I'd expect to work without optimizing the absolute heck out of things. (i'm talking about using Direct-to-wire for streaming, few/no group streams, etc, low frame rates). If it is more than 16 then ... good luck.

 

[uknod]

Thanks for all the answers I will answer them all now in one go:

Have you checked your network's bandwidth ? (speed / capacity)

Is the hardware 10/100 Megabit?

It is I gb. Cameras are all hard wired, web logins are all local wireless, there are no botlenecks on the wireless as if I open up a web logon on the main PC then that has the same issue.

Also, suggest using resource monitor to look carefully at how the HDDs are working, especially the active time and the latency.

I am not recording anything, I turned that off to eliminate and hard drive probles, but it is using a solid state drive with no load on it at all.

Are the cameras Wifi? Are the login connections wifi ? Provide a network drawing.

No the cameras are hard wired. Attached drawing

Just one Neolithic cavemans observation.
One bottleneck I can think of is using 1 hard drive. I'd be using multiple hard drives for that much thruput.
It may not be ALL of your issue, but in my experience my Blue iris systems are more responsive when the hard drives are not bogged down.
I guess i dont understand the " web logins" portion of the problem, but if your viewing 16 cameras on 16 seperate internet streams you'd need Fiber internet or a very fast copper internet with high Up/down bit rates

Already answered, I am not recording so very little hard drive use. Also for you subsequent question, I am using the latest version of BI, even tried older vertions to make sure.

I don't think dual processor systems are supported well by Blue Iris (or most other software for that matter). If CPU load is even close to 50%, that is more like 100%. I don't think Blue Iris can go meaningfully over 50% on a dual CPU system, is what I'm saying.

Also, if it is a dual processor Xeon, that probably means it is 10+ years old (otherwise you likely wouldn't have been able to afford it), and older CPUs aren't very fast no matter how many cores they have. What CPU model is it exactly?

You say "16 web connections to view each individual camera" but what are those connections to? To Blue Iris's web server? You have 16 connections to each of 16 cameras (16x16 = 256)??? Can you clarify this? Because even 16 connections is more than I'd expect to work without optimizing the absolute heck out of things. (i'm talking about using Direct-to-wire for streaming, few/no group streams, etc, low frame rates). If it is more than 16 then ... good luck.

In the words of BI support when they took a look for me, "wow you have some serious hardware there", so CPU is not a problem, but just in case, I decided to put in a completely new PC using the latest I7 cpu with the I7's graphical acceslerator thinking this may be the problem, result was it could just about manage it all on substeream, but cpu was bouncing off of 100%, and one camera switched to main stream killed it. So CPU is very good. They are xeon 5s, so not the latest but still pretty awsome.
Not using internet, all the web connections are local, but the ui3 is pretty good with the bandwidth. Once all the streams are running you can even duplicate streams on more device and it doesn't seem to effect any of the bandwidths, which I thought was pretty clever. (ie. One camera connecting to 5 seperate web streaming devices, only the first one uses up some hardware resource, the other 4 are neglidgable)

 

[TonyR]

I'm not familiar with the symbol at the top although it resembles a "Wi-Fi" symbol.......is that a router?
If it is, do you have streaming video coming from 16 cameras via the server connected to a router's LAN port?

 

[uknod]

I'm not familiar with the symbol at the top although it resembles a "Wi-Fi" symbol.......is that a router?
If it is, do you have streaming video coming from 16 cameras via the server connected to a router's LAN port?

Yes its wifi, sorry for my poor artwork. So all web connected devices are wifi. But there is no sign of any strangling on the mesh wifi logs, also if I connect a web connection on the actual server pc, then that also has the same issues.

 

[TonyR]

Yes its wifi, sorry for my poor artwork. So all web connected devices are wifi. But there is no sign of any strangling on the mesh wifi logs, also if I connect a web connection on the actual server pc, then that also has the same issues.

I think a bigger issue is that router LAN and WLAN (wireless) ports are not built to handle constant video streams, especially from 16 cameras and especially at high-res.

 

[wittaj]

+1 above!

Cameras connected to Wifi routers (whether cams are wifi or not) are problematic for surveillance cameras because they are always streaming and passing data. And the data demands go up with motion and then you lose signal. A lost packet and it has to resend. It can bring the whole network down if trying to send cameras through a wifi router. At the very least it can slow down your entire system.

Unlike Netflix and other streaming services that buffer a movie, these cameras do not buffer up part of the video, so drop outs are frequent, especially once you start adding distance or number of connections. You would be amazed how much streaming services buffer - don't believe me, start watching something and unplug your router and watch how much longer you can watch NetFlix before it freezes - mine goes 45 seconds. Now do the same with a camera connected to a router and it is fairly instantaneous (within the latency of the stream itself)...

The same issue applies even with the hard-wired cameras trying to send all this non-buffer video stream through a router. Most consumer grade wifi routers are not designed to pass the constant video stream data of cameras, and since they do not buffer, you get these issues. The consumer routers are just not designed for this kind of traffic, even a GB speed router.

 

[uknod]

+1 above!

Cameras connected to Wifi routers (whether cams are wifi or not) are problematic for surveillance cameras because they are always streaming and passing data. And the data demands go up with motion and then you lose signal. A lost packet and it has to resend. It can bring the whole network down if trying to send cameras through a wifi router. At the very least it can slow down your entire system.

Unlike Netflix and other streaming services that buffer a movie, these cameras do not buffer up part of the video, so drop outs are frequent, especially once you start adding distance or number of connections. You would be amazed how much streaming services buffer - don't believe me, start watching something and unplug your router and watch how much longer you can watch NetFlix before it freezes - mine goes 45 seconds. Now do the same with a camera connected to a router and it is fairly instantaneous (within the latency of the stream itself)...

The same issue applies even with the hard-wired cameras trying to send all this non-buffer video stream through a router. Most consumer grade wifi routers are not designed to pass the constant video stream data of cameras, and since they do not buffer, you get these issues. The consumer routers are just not designed for this kind of traffic, even a GB speed router.

Sorry if I wasn;t clear. The cameras do not go through any routers, they are hardwired to a gigbyte poe switch along with the BI pc. The only thing going through the wireless is the web ui3 logons, which as I have said above is also effected on the main PC, so not a wireless issue. If you think a GB switch would also have this problem, then I will need to investigate this, but I doubt it is the issue.

 

[uknod]

I think a bigger issue is that router LAN and WLAN (wireless) ports are not built to handle constant video streams, especially from 16 cameras and especially at high-res.

Is this the case if it does not go through router, as the cameras only go through the LAN GB switch.

 

[TonyR]

Is this the case if it does not go through router, as the cameras only go through the LAN GB switch.

Since I saw none on the drawing I assumed the BI server was connected to a LAN port on the wireless router....and that too can be an issue.