Add OpenVino Detector #3768
Add OpenVino Detector #3768
Conversation
NateMeyer
force-pushed
the
ov-detector
branch
from
c3135a9
to
da89bb5
Compare
|
I have two questions with this:
|
I know a couple users have mentioned they have one, not sure which ones will be able to test, you might want to make a discussion post about it.
It's not clear to me what you're asking. Are you saying the multiarch build fails somewhere? Are all the dependencies needed for the VPU in the ov package that is added? If not then I'd imagine you'd just need to add the |
The intel-opencl-icd package is only needed for running on the Intel GPU, and is only available for the amd64 arch. The multiarch build is failing for me, I assume it is just a setup issue on my end. Are there some instructions for running the multiarch build? My quick googling looks like I need to setup qemu on my system to run the ARM builds. I think the openvino python library should have the necessary pieces for using the VPU plugin, except for the driver which would have to be installed on the host system. I would then expect the enumerated VPU device could be passed through to the container. |
|
You will want to use https://github.com/blakeblackshear/frigate/blob/release-0.11.0/Makefile#L29 to build multiarch. If there is an error then you'll need to paste it here and I can advise. |
|
The error I get is: when it gets to the first It is complaining that it can't run the arm binaries in the arm image. I just installed qemu and followed the setup at https://github.com/multiarch/qemu-user-static and got past this first error. To prove out my build system, I built the release-0.11.0 branch. This successfully built each architecture. Finally I had to create a buildx builder to create the multiarch image at the end with Back on my openvino branch, now I get to the next issue. The Python package available from pypi only support amd64 architecture. In order to support OpenVino on arm, it looks like we'll have to add another stage to the build to compile the runtime from source, until someone publishes a python package. The repository from Intel will build on ARM, but only supports the VPU device. OpenVino Build Instructions for RaspbianOS There is another plugin that supports the ARM CPU for OpenVIno that is maintained by OpenCV, but I'm not sure that will bring much value to this project. OpenVino ARM CPU Plugin Repo At this point, I think I'll keep this to amd64 only. If there is strong desire to add the build of the openvino runtime on arm, we can look at adding it then. |
|
Yeah, the NCS2 is already end-of-lifed so I don't know if that makes sense to worry about. |
|
By the way what inference times are you seeing? |
|
The model benchmark in the OpenVino toolbox claims my GPU will do 263fps with this model. Looks like the debug page is showing an inference speed of 8-12ms. This is just measuring latency? My GPU is barely loaded during this, but I am only running with a single camera at the moment. The measure of throughput could be higher than the inference speed suggests if the async requests or batching are played with. |
|
The debug page is the average amount of time it has taken to start the detection then get a result. Maybe with more cameras it could be loaded higher but that is quite good. |
|
Just wanted to say I tested this on my i5-4590 using the CPU plugin and I'm getting 15-18ms with 10-30% CPU usage (2 cores on a Proxmox LXC container). Much better than the built-in CPU plugin, I would get around 80-100ms (sometimes as high as 200ms) and 80% CPU usage. I believe my iGPU is unsupported so I can't test that (plus I think it's disabled due to having a dedicated GPU installed). This plugin gets about the same, if not slightly better, speed as the TensorRT CUDA plugin with my P400 card (#2548 / #3016). I definitely hope this gets merged into Frigate, awesome job on this! |
Assuming you're running on linux it doesn't work that, they'll both work just fine together. What error were you getting when trying it with the GPU? @NateMeyer We will definitely need documentation on which intel iGPUs are supported for this. |
|
Yea, it should support GPUs per Intel's "Supported Devices" documentation but we can be explicit about it too. This still needs to either add ARM support for openvino, or add more platform checks in the detector code. As it is, it will crash on the arm64 and armv7 builds. |
If I use the OpenVIno hello_device_query.py sample it only shows CPU, no GPU device I also tried running one of OpenVino's docker images that have all the runtimes and drivers installed and I get the same result. It looks like Intel NEO Compute compatibility is required and that is Broadwell and above (https://github.com/intel/compute-runtime#supported-platforms) |
|
@yeahme49 Perfect thanks, that will be helpful for our docs. |
|
@NateMeyer Looks like we'll want to update the docs that intel GPUs are supported 5th gen and up. |
|
Intel's documentation looks like 6th gen and up. I'll get this info added after we settle out the NCS2 and ARM support. |
Sounds good, seems like you're making headway with it. |
|
Hello, I have a nsc2. I guess I can make a test. |
Hi @titilambert, there are some instructions in the discussion topic #3797. Just add the detector and model blocks to your normal config, and point to my test image. Post a comment over there and I can help with any other questions. Thanks! |
|
@NateMeyer, thank you very much for your work! I have been waiting to be able to use OpenVINO with Frigate, since in my country it is impossible to get a Google Coral USB. Also, previously using the OpenVINO object detection demo I noticed that nanodet models were not giving me the false positives that I am having with ssdlite. As I wanted to test nanodet with Frigate but I don't know anything about how OpenVINO or Frigate works internally I modified the code to add the Model API that is in the Open Model Zoo of OpenVINO, as it abstracts the adaptation of the tensor to the model and process its output. No doubt it is inefficient, at least because although the tensor_transform function explicitly says that only works permutations B, H, W, C the nanodet models need B, C, H, W and yet when passing the tensor to the Model API model adapter the inference is performed correctly anyway. Even with the very anemic GPU of a Celeron J4025 I have a 65-70 ms latency with nanodet-plus-m-1.5x-416 (with nanodet-m-1.5x-416 I had a slightly lower latency, 50-55 ms), using OpenVINO 2022.2.0. In case anyone is interested in trying nanodet in this gist there are the diffs of the modifications I made, in config.yml you pass model_type as "NanoDet", "NanoDetPlus" or "SSD" and you have to mount openvino-model externally in docker-compose.yml to pass the corresponding IRs (NanoDet uses a different tag file, coco_80cl.txt). I tried to modify Dockerfile to perform the conversion and add it to the Docker image, but when installing the OpenVINO Python dependencies to perform the Pytorch model conversion it starts to crash because of missing libraries (libGL). I think it is a great advantage of Frigate to support OpenVINO because besides supporting several generations of iGPUs the latest version announces Intel Arc support. |
|
Hi @CarlosML, I'm glad you've found it useful! I played with a few different models when I was working on this but I don't think I tried NanoDet. My intent with this detector would be that you convert a model to the IR format, and you can configure the input format in the config (BCHW vs BHWC, etc). I haven't yet added configurable processing on the detection output. You've worked around this with using the Model API from OMZ. This looks like a neat API, I hadn't considered using it before. Ideally we could have the shape of the results be configurable and generic enough that the description in the model config would apply to different detector frameworks (i.e. Tensorflow-lite, RKNN, etc.). I'm open to suggestions on how that would look. SSD, NanoDet, and Yolo have fairly different output tensors that need to be parsed a bit differently. Perhaps someone could add a model-parse utility with common functions to get labeled detection boxes from the various different model types. I think these could be done in a detector-agnostic way. |
|
One issue that I don't know how important is that Frigate by default uses SSDLite Mobiledet which I understand has a higher accuracy score than MobileNetV2+SSDLite which is in OMZ. I understand this one would be similar to the one Frigate downloads from the Coral repository (minus some post processing), late last year I tried to convert it with OpenVINO and it failed and still fails 'automatically' with the latest version but the suggestion I was given in the OpenVINO repository seems to work fine. Here is the output of mo.py if anybody is interested in trying it. |
True, but this croud funding campaign just launched which is based on the same processer. |
Nevermind, seems it is going to be able to support the same SDK |
|
Intel says they are deprecating the NCS2 support in OpenVino after 2022.3
https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/system-requirements.html |
|
You're installing those from the testing repo? What version is getting installed? This is what I see in the current image $ docker run ghcr.io/natemeyer/frigate:0.12.0-openvino-8230b34 apt list --installed | grep intel-
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
intel-media-va-driver-non-free/now 21.1.1+ds1-1 amd64 [installed,local]
intel-opencl-icd/now 20.44.18297-1 amd64 [installed,local]
|
|
Looks good, we'll definitely want to update the docs recommended hardware to users know that these options exist, but I think it makes sense to wait given Nvidia Tensor devices will probably be added to that as well. Only other thought is might want to look at #4395 and see if that can also work for the stages that are being added in this PR |
|
@EduardoLaranjo I believe the updated driver support will be included in #4368 |
|
@NateMeyer this will need to be rebased on dev since there have been a bunch of docker changes. Should make for a cleaner build though |
Yes, I've been watching for it to settle out. Are there any more issue or changes for the build/docker/dev env needed or should it be ready now? |
I think there may be a few stragglers but I think (hope) the structure should be finished and no more organizational changes |
NateMeyer
force-pushed
the
ov-detector
branch
2 times, most recently
from
1e975b9
to
fdcc476
Compare
|
Rebased. Built a new test image at |
Sucessfully load model and initialize OpenVino engine with either CPU or GPU as device. Does not parse results for objects.
- Use custom-built openvino wheel for all platforms - Add libusb build without udev for NCS2 support
NateMeyer
force-pushed
the
ov-detector
branch
from
fdcc476
to
004ebc5
Compare
|
@NateMeyer love your work, this has been amazing on my intel thin client for lowering power and CPU usage. One thing that has plagued me is that ssdlite_mobilenet_v2 seems to be causing a ridiculous number of false positives picking up a a firepit and some pot plants as human, but at such a high percentage that I can't eliminate it with thresholds (see below):
Any guides or instructions on what I need to change in the Dockerfile to test another model (I understand from the above that anything with the same post-processing can just be swapped out) - I assume by simply changing the omz-download and omz-converter lines. |
|
Is there any outstanding work for this PR, or is it ready to merge? Have you tested a multiarch build? |
I think this is ready. The only outstanding work may be tweaking the included model, as @nickp27 points out above. I have been building the multiarch image, but I've only run it with qemu personally (amd64, arm64, and arm32). There was some work in #3797 to run it with the NCS2 stick on amd64 and arm64. The image seemed to work ok on arm64, but I think we were running into some issues with the NCS2 on kubernetes.
@nickp27, Clearly you need to move your plants to not be so lifelike 😆 . I'll dig up my notes on which models dropped in. If this gets merged I think you can open a regular issue and may get other suggestions about masking it out, I'm not as familiar myself with creating zones. Are you thinking of rebuilding the image or you can run just the ov-converter stage and mount the converted model in the prebuilt image. |
|
@nickp27 here are a list of some models I think will drop in from the Intel OMZ.
These models may have different sizes and labelmaps that also need to have the correct adjustments in the model config. Also, I didn't have much luck with the ssd300 model, I didn't debug why I wasn't getting detection boxes even though the debug log seemed to have detections. Others may be the same. I wanted to try a nanodet or yolo model, but this work was left for another time. |
|
@NateMeyer Thanks for that! I tried converting yolov4-tiny to match the development work being done on tensorrt but haven't figured out how to parse the multiple outputs just yet. Will give a combination of new models and some filters a go! |
|
not sure if this is the right place, just to provide some feedback, cpu util dropped from 60% (at most, had only 4 cores allowed in detectors) to 25% (at most) inference speed dropped from 100ms to 75ms thanks a lot for this ! :-) no need for coral nor machine upgrade for now |
This PR is dependent on the Object Detector Abstraction change #3656.
This adds the OpenVino framework and a default model to the build image and a detector implementation that uses the OpenVino framework. I have developed and tested this on my i5 1135G7, using the Xe iGPU.
The necessary configuration to run the included model is explained in the Detectors page of the documentation.
TODO:
- Update default modelFurther Model support can be reviewed/developed later