Poor AI Detection Speed

bethzur

Getting the hang of it
May 19, 2021
103
57
Pennsylvania
I have an RTX 3050 in my PC with an i7-7700. I see AI detections take 500-1300 ms. That seems longer that it should be by at least an order of magnitude. I checked the logs and it claims it's using the GPU. I'm using yolov8s for the model. I seem to recall DeepStack was 100 ms or less. Has anyone else seen this or any tips to troubleshoot. I may try 8n which I'm guessing should be faster. I am using the main stream, so the images are like 2880x1530 but Google AI thinks that should be around 100 ms.

Here is a sample from the logs. There are no refrigerators in the image.

Code:
T-767 msec  [1446 msec]   
[
    {
        "api":"primary",
        "found":{
            "success":true,
            "error":null,
            "message":"",
            "predictions":[
                {
                    "category":4,
                    "label":"refrigerator",
                    "confidence":0.718442559242249,
                    "x_min":1095,
                    "x_max":1451,
                    "y_min":411,
                    "y_max":799}
                ,
                {
                    "category":4,
                    "label":"skis",
                    "confidence":0.449722796678543,
                    "x_min":1687,
                    "x_max":2030,
                    "y_min":834,
                    "y_max":1115}
                ,
                {
                    "category":4,
                    "label":"refrigerator",
                    "confidence":0.441071748733521,
                    "x_min":640,
                    "x_max":1188,
                    "y_min":3,
                    "y_max":1125}
                ,
                {
                    "category":4,
                    "label":"refrigerator",
                    "confidence":0.415100038051605,
                    "x_min":1077,
                    "x_max":1341,
                    "y_min":772,
                    "y_max":1040}
                ]
            ,
            "count":4,
            "moduleId":"yolov8s.onnx",
            "executionProvider":"ONNXRunTime: 3 (NVIDIA GeForce RTX 3050)",
            "analysisRoundTripMs":1015}
        }
    ]
 
I have an RTX 3050 in my PC with an i7-7700. I see AI detections take 500-1300 ms. That seems longer that it should be by at least an order of magnitude. I checked the logs and it claims it's using the GPU. I'm using yolov8s for the model. I seem to recall DeepStack was 100 ms or less. Has anyone else seen this or any tips to troubleshoot. I may try 8n which I'm guessing should be faster. I am using the main stream, so the images are like 2880x1530 but Google AI thinks that should be around 100 ms.

Here is a sample from the logs. There are no refrigerators in the image.

Code:
T-767 msec  [1446 msec]  
[
    {
        "api":"primary",
        "found":{
            "success":true,
            "error":null,
            "message":"",
            "predictions":[
                {
                    "category":4,
                    "label":"refrigerator",
                    "confidence":0.718442559242249,
                    "x_min":1095,
                    "x_max":1451,
                    "y_min":411,
                    "y_max":799}
                ,
                {
                    "category":4,
                    "label":"skis",
                    "confidence":0.449722796678543,
                    "x_min":1687,
                    "x_max":2030,
                    "y_min":834,
                    "y_max":1115}
                ,
                {
                    "category":4,
                    "label":"refrigerator",
                    "confidence":0.441071748733521,
                    "x_min":640,
                    "x_max":1188,
                    "y_min":3,
                    "y_max":1125}
                ,
                {
                    "category":4,
                    "label":"refrigerator",
                    "confidence":0.415100038051605,
                    "x_min":1077,
                    "x_max":1341,
                    "y_min":772,
                    "y_max":1040}
                ]
            ,
            "count":4,
            "moduleId":"yolov8s.onnx",
            "executionProvider":"ONNXRunTime: 3 (NVIDIA GeForce RTX 3050)",
            "analysisRoundTripMs":1015}
        }
    ]
Do not use main stream, it does not improve accuracy, it just slow down the detection.
 
It may be using the iGPU instead of your Nvidia GPU, try disabling the iGPU in the BIOS and see if that changes the speed
 
I did not. Where is that?
Read this post for location and instructions that I posted earlier!
 
  • Like
Reactions: bethzur