CodeProject.AI 2.9.5 w/YOLOv5.NET (GPU/DirectML) memory growth/leak

C

carteriii

Pulling my weight
Jan 8, 2016
164
173
USA
Running on Windows 11 Pro, running about 40 streams through CP.AI, after enough time & events, the memory use of the CodeProject.AI Server through Task Manager still shows only ~150MB on the process tab, BUT the total memory use shown on the performance tab grows over time to 10-15GB at which point I either have to restart the service or bad things happen. When I use the services window to restart "CodeProject.AI Server", the system memory use drops by that 10-15GB, indicating that the memory lost is within CodeProject.AI. Here are two screenshots showing the stated memory usage, and then the memory usage after stopping the CodeProject.AI Server after about 440k calls to the Object Detection (v1.14.0), and 77k calls to the License Plate Reader (3.3.4).
bi-cpai-process-memory-use.jpg
cpai-memory-drop.jpg

Having said that, I realize it's possible this is due to BlueIris' interaction with it, perhaps not freeing up connections, BUT then I would expect the process memory use to show growth as well.

The 10GB+ memory growth typically happens over the course of a week or more, but when the winds are blowing causing lots of false activations, it can happen in just a few days. At a minimum, it does seem directly proportional to the number of calls to CP.AI. I have a second instance of BI with only 4 cameras and never see this kind of growth.

Has anyone else experienced this? Are there specific lines in the CP.AI log file that I can try to match up (e.g. new connection call vs. free'd up connection or call) that might help identify or at least narrow down the root cause of this? I'm not asking anyone to do the work for me, but please help steer me in a direction that might result in a finding that can be useful to everyone. Thanks in advance.
 
I've had some incredible instability lately and made a post on it here:

kjinxx2

Thread 'Very Odd CodeProject.AI/Blue Iris Behavior - Thoughts?'

Hi all - running BI 5.9.9.24 (latest stable update) and CodeProject.AI 2.9.5.0

I don't know when/why this started - but BlueIris is randomly receiving TONS of positive identifications on all of my cameras from CodeProject.AI. Below is just an example, this happens on all of my cameras. I've tried reinstalling CPAI, updating BI (this started happening last stable release), etc. I've seen this issue before where multiple models are triggering, but I've configured BI & CPAI to only use one model.

Below is a picture of the alert image & AI Debugger...

Any thoughts on how I can debug this...

I also am experiencing memory leakage. Every few days CPAI completely stops responding to BI and I must restart CPAI through the BI interface (or restart the machine).

I never solved this issue
 
I'd like to add to this, that this is still an ongoing, and major issue. I have tried all number of combinations to resolve this and none seem to resolve it. My default setup is running CPAI in a docker container on unraid, but I have tried directly in a windows install as well with no change.

In my testing, I have found that no model seems to make any difference, Yolo v6, Yolov8, Yolo .net variants all experience the issue the same, and regardless of GPU (nvidia in my case) or CPU processing, the leak still occurs. I've tested this across multiple servers and the results are the same.

There have been no updates to CPAI in over 8 months, and this issue just cropped up about a month or so ago, maybe less, with no change to CPAI at all, so the issue seems to be in the way Blue Iris is sending requests to CPAI and there's no garbage collection being done. I'm not sure if requests are being sent a new way or something, but from all the variables I've considered and tested, this is a breaking change from Blue Iris's side, not CPAI.

Have messaged support about this and awaiting a response. Hopefully can get this resolved soon as ballooning from 4gb to 15+ and never stopping is hugely problematic lol.
 
  • Like
Reactions: prsmith777
I found the issue to be related to the static object detection (which would prevent movement in a camera from triggering an alert if, say, a car was parked in your driveway and not moving but a shadow triggered movement, etc). I see that in the latest patch this has all been reworked to a 'periodic scan' feature but I haven't yet been able to mess with it. Disabling the 'static object detection' on all of my cameras resolved the issue for me.
 
prsmith777 said:
Also noticing this. My bandaid solution is to reboot every few days
Click to expand...
Agreed. I first saw this last week I think. It was disconcerting because I just stopped getting alerts, with no good way to monitor the behavior.
For a while, I was rebooting the BI machine weekly.
But at one time, I got close to one million inferences.
Same 2.9.5 version.
I didn't realize it has been 8 months.
It's starting to look like abandonware to me.
Having good luck with Frigate on Ubuntu with a Coral USB accelerator.
(I also miss Deepstack)

Sent from my iPlay_50 using Tapatalk
 
  • Like
Reactions: kjinxx2
hapstabu said:
Agreed. I first saw this last week I think. It was disconcerting because I just stopped getting alerts, with no good way to monitor the behavior.
For a while, I was rebooting the BI machine weekly.
But at one time, I got close to one million inferences.
Same 2.9.5 version.
I didn't realize it has been 8 months.
It's starting to look like abandonware to me.
Having good luck with Frigate on Ubuntu with a Coral USB accelerator.
(I also miss Deepstack)

Sent from my iPlay_50 using Tapatalk
Click to expand...

Yes this was the exact same behavior that I experienced.
 
hapstabu said:
Agreed. I first saw this last week I think. It was disconcerting because I just stopped getting alerts, with no good way to monitor the behavior.
For a while, I was rebooting the BI machine weekly.
But at one time, I got close to one million inferences.
Same 2.9.5 version.
I didn't realize it has been 8 months.
It's starting to look like abandonware to me.
Having good luck with Frigate on Ubuntu with a Coral USB accelerator.
(I also miss Deepstack)

Sent from my iPlay_50 using Tapatalk
Click to expand...
Since this issue has cropped up, I have been tinkering and debating switching to a Frigate motion detection system and completely abandoning CPAI, as, like you said, it seems to be becoming abandon ware. Have had some inital success using it, but haven't had the time to fully make an inference setup with proper blacked out areas and tuning. how's your experience so far? I'm dreading manually configuring all 9+ of my cameras in YAML, but I've done it in HA before so, just need to try to get one fully setup to see if the effort is justified.

I liked the idea that CPAI had, and it was nice being able to choose your models and other things in a webui, but I've ran into issue after issue with it, and almost wish blue iris would roll their own solution using existing models or integrate with frigate directly, since after all, it is just YAML.

"Glad" to see I'm not the only one suffering, I thought I was going crazy for a while!
 
TheOgre said:
Since this issue has cropped up, I have been tinkering and debating switching to a Frigate motion detection system and completely abandoning CPAI, as, like you said, it seems to be becoming abandon ware. Have had some inital success using it, but haven't had the time to fully make an inference setup with proper blacked out areas and tuning. how's your experience so far? I'm dreading manually configuring all 9+ of my cameras in YAML, but I've done it in HA before so, just need to try to get one fully setup to see if the effort is justified.

I liked the idea that CPAI had, and it was nice being able to choose your models and other things in a webui, but I've ran into issue after issue with it, and almost wish blue iris would roll their own solution using existing models or integrate with frigate directly, since after all, it is just YAML.

"Glad" to see I'm not the only one suffering, I thought I was going crazy for a while!
Click to expand...
If you are going to try Frigate, use Microsoft Visual Studio Code on your Linux machine.

And there is a way to point to the yaml-language-server in the Frigate API. Put this in the first line:

#yaml-language-server:$schema=http://###.###.###.###:5000/api/config/schema.json

This helps a lot to figure out the language (?) of YAML.

It gives you the options as you type.

My experience is that indentation is really important in YAML.



Sent from my iPlay_50 using Tapatalk
 
TheOgre said:
Since this issue has cropped up, I have been tinkering and debating switching to a Frigate motion detection system and completely abandoning CPAI, as, like you said, it seems to be becoming abandon ware. Have had some inital success using it, but haven't had the time to fully make an inference setup with proper blacked out areas and tuning. how's your experience so far? I'm dreading manually configuring all 9+ of my cameras in YAML, but I've done it in HA before so, just need to try to get one fully setup to see if the effort is justified.
Click to expand...

You tune the masking etc in the UI, so its not hard at all
 
prsmith777 said:
Also noticing this. My bandaid solution is to reboot every few days
Click to expand...
Happened again twice in the last 48 hours or so.
I checked the alerts and the data files associated with them.
Dat files show all the images, but returning nothing found from the model that was used.
If I pass the image to CPAI with the model after restarting the object detection, it is identified immediately.
But before restarting the object detection, passing a test image (from CPAI) finds nothing.
This happens with multiple object detection methods.
What puzzles me is that I have had ver. 2.9.5 running since December 2024 with no apparent issues.
I received no updates from Windows this month, but last month I received KB5063878 on 8/15/2025.
Maybe that broke something.
I now have the machine set to reboot weekly, but I saw this happen twice in so many days.
The next time I see this, I will take care to check the memory usage before I start it.
BTW, the first time it quit, it had over one million inferences.
 
hapstabu said:
Happened again twice in the last 48 hours or so.
I checked the alerts and the data files associated with them.
Dat files show all the images, but returning nothing found from the model that was used.
If I pass the image to CPAI with the model after restarting the object detection, it is identified immediately.
But before restarting the object detection, passing a test image (from CPAI) finds nothing.
This happens with multiple object detection methods.
What puzzles me is that I have had ver. 2.9.5 running since December 2024 with no apparent issues.
I received no updates from Windows this month, but last month I received KB5063878 on 8/15/2025.
Maybe that broke something.
I now have the machine set to reboot weekly, but I saw this happen twice in so many days.
The next time I see this, I will take care to check the memory usage before I start it.
BTW, the first time it quit, it had over one million inferences.
Click to expand...
Did you try what I suggested?
 
kjinxx2 said:
Did you try what I suggested?
Click to expand...
Actually, Periodic scan was already off, static object detection was also off for a long time, I had problems with it almost a year ago, and had never turned it back on. I did work on reducing nothing found inferences by masking some distant areas to keep from triggering on motion in the distance, especially at night. I also reduced the number of images sent by changing the timing.
That has reduced the number of images sent to CPAI. I am rebooting the machine weekly. I also rolled black to 5.9.9.59 Blue Iris. Since no changes were made in the last several months. So far so good. Still working for a few days.
 
Failed again Thursday morning.
Switched to ObjectDetectionCoral, but the accuracy is so bad, that I just can't use it.
I fired up Docker ver 2.9.7 image on a Ubuntu machine. It's running but terribly slow. Like 1.5 seconds per inference. But it is functional at this time using Object Detection YOLOv5 6.2. I will see if it stays running for a while on Linux. I will also reach out to Ken to ask if he has heard of this and if he has heard anything from the CPAI team. I will also post on the CPAI GitHub site, but I don't see any responses there for a while.
 
CPAI is dead for all intents and purposes, at least from my perspective. Another alternative must be implemented as soon as possible
 
Yes. Windows 11 for Blue Iris. Ubuntu 24.0x for Frigate NVR, Nagios and a couple of other things.
Currently on the Windows machine, CPAI version 2.9.7 image on Docker Desktop using WSL Ubuntu back end. Much faster than the other Linux machine. Seeing if it is more reliable.
Moving away from Windows for everything else.
 
You must log in or register to reply here.
Top Bottom