Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

convert yolov5 to openvino #891

Closed
linhaoqi027 opened this issue Sep 1, 2020 · 48 comments · Fixed by #6057
Closed

convert yolov5 to openvino #891

linhaoqi027 opened this issue Sep 1, 2020 · 48 comments · Fixed by #6057
Labels

Comments

@linhaoqi027
Copy link

linhaoqi027 commented Sep 1, 2020

Before You Start

train your yolov5 model on your own dataset following Train Custom Data

  • change Activation Function to Relu or Leaky ReLU, because Hardswish is not suported in onnx and openvino.Pre-trained model can be loaded ignoring which Activation Function u use.
    self.act = nn.Hardswish() if act else nn.Identity()

export model to onnx,following #251

before run export.py:

  • changeto torch==1.5.1,torchvision==0.6.1
  • change

    yolov5/models/yolo.py

    Lines 49 to 53 in 5e0b90d

    y[..., 0:2] = (y[..., 0:2] * 2. - 0.5 + self.grid[i].to(x[i].device)) * self.stride[i] # xy
    y[..., 2:4] = (y[..., 2:4] * 2) ** 2 * self.anchor_grid[i] # wh
    z.append(y.view(bs, -1, self.no))
    return x if self.training else (torch.cat(z, 1), x)
                c=(y[..., 0:2] * 2. - 0.5 + self.grid[i].to(x[i].device)) * self.stride[i]  # xy
                d=(y[..., 2:4] * 2) ** 2 * self.anchor_grid[i]  # wh
                e=y[..., 4:]
                f=torch.cat((c,d,e),4)
                z.append(f.view(bs, -1, self.no))

        return x if self.training else torch.cat(z, 1)
model.model[-1].export = False
  • set

    yolov5/models/export.py

    Lines 51 to 52 in 5e0b90d

    torch.onnx.export(model, img, f, verbose=False, opset_version=12, input_names=['images'],
    output_names=['classes', 'boxes'] if y is None else ['output'])

    to opset_version=10.Because only opset=10 support resize ops.

then you can run export.py to export onnx model.

install OPENVINO2020R4

you can install following https://bbs.cvmart.net/topics/3117

convert onnx to openvino through

python3 /opt/intel/openvino_2020.4.287/deployment_tools/model_optimizer/mo.py \
--input_model {input_dir}/yolov5s.onnx \
--output_dir {output_dir}  \
--input_shape [1,3,480,480]

then you can get openvino model .bin and .xml.

WARNING:NMS is not included in openvino model.

AS for how to use openvino to inference ,please click my profile or https://github.com/linhaoqi027/yolov5_openvino_sdk

@0806gcx
Copy link

0806gcx commented Sep 2, 2020

When I changed the py file as you said, this error occurred. After I change the yolo.py file, do I need to retrain the networkmodel?
image

@linhaoqi027
Copy link
Author

When I changed the py file as you said, this error occurred. After I change the yolo.py file, do I need to retrain the networkmodel?
image

NO,u don't need to retrain the model. mabe you can change e=y[...,4:6] to e=y[..., 4:] .I write e=y[...,4:6] because I just have one class. Please tell me if it works so i can solve the bug.

@0806gcx
Copy link

0806gcx commented Sep 2, 2020

When I changed the py file as you said, this error occurred. After I change the yolo.py file, do I need to retrain the networkmodel?
image

NO,u don't need to retrain the model. mabe you can change e=y[...,4:6] to e=y[..., 4:] .I write e=y[...,4:6] because I just have one class. Please tell me if it works so i can solve the bug.

well,thank you well ,I have successfully converted, but will you encounter this warning?
image

@linhaoqi027
Copy link
Author

When I changed the py file as you said, this error occurred. After I change the yolo.py file, do I need to retrain the networkmodel?
image

NO,u don't need to retrain the model. mabe you can change e=y[...,4:6] to e=y[..., 4:] .I write e=y[...,4:6] because I just have one class. Please tell me if it works so i can solve the bug.

well,thank you well ,I have successfully converted, but will you encounter this warning?
image

ignore it

@lucasjinreal
Copy link

Have u guys tried:

python -m onnxsim a.onnx a_sim.onnx

to simplify your onnx model? This is really helpful to eliminate unsupported ops and simplify whole onnx model structure.

If you guys can simplify your onnx model pls let me know.

@0806gcx
Copy link

0806gcx commented Sep 2, 2020

Excuse me ? how do i change Activation Function to Relu or Leaky ReLU, I can't find the corresponding function
image

@linhaoqi027
Copy link
Author

打扰一下 如何将激活功能更改为Relu或Leluy ReLU,我找不到对应的功能
图片

it is in

self.act = nn.Hardswish() if act else nn.Identity()
.
change it before train. Then using pre-trained model or self-train directly.

@0806gcx
Copy link

0806gcx commented Sep 3, 2020

I did it your way, trained a new network, but encountered new problems when I converted onnx to vino
image

@lucasjinreal
Copy link

lucasjinreal commented Sep 3, 2020

@0806gcx As I mentioned above, the CPU mode traced ONNX model somewhat hard to convert to other framework such as TensorRT and OpenVINO. It can not even pass test in onnxruntime. @glenn-jocher

Simply verify it with onnx-simplifier.

@0806gcx
Copy link

0806gcx commented Sep 3, 2020

@jinfagang
But this blogger successfully launched on openvino,I need to do it well.

@lucasjinreal
Copy link

@0806gcx It says model have a concat error in a Cocat node. dimension not match.

@lucasjinreal
Copy link

lucasjinreal commented Sep 3, 2020

@0806gcx I think @glenn-jocher Add something and break this model, I previous using v1.0 and v2.0 can also sucessfully export onnx and convert to tensorrt very quickly, and now it breaks. Since this repo's author doesn't care much about cuda side (he always export in cpu mode) so that he doesn't realize this bug. But indeed, v3.0 exported onnx doesn't work any longer, it has some wrong Concat connnection and I can not even export it successfully in CUDA mode.

@linhaoqi027
Copy link
Author

I did it your way, trained a new network, but encountered new problems when I converted onnx to vino
image

I met similar bug ever. DId you use openvino2020R4?

@linhaoqi027 linhaoqi027 changed the title convet yolov5 to openvino convert yolov5 to openvino Sep 3, 2020
@FrzWang
Copy link

FrzWang commented Sep 3, 2020

I met similar bug too. using openvino2020R4+LeakyRelu()
Screenshot from 2020-09-03 22-57-22

@linhaoqi027
Copy link
Author

I met similar bug too. using openvino2020R4+LeakyRelu()
Screenshot from 2020-09-03 22-57-22
Did your version is torch==1.5.1,torchvision==0.6.1 when convert torch to onnx?Higher version may cause problem(This problem will cause error when convert onnx to openvino)

@glenn-jocher
Copy link
Member

@linhaoqi027 @0806gcx @jinfagang unfortunately I don't have any openvino experience so I can't help here, but I think it would be worth raising this issue directly on the openvino repository since the errors are generated there.

The scope of the YOLOv5 repository as it stands now is limited to basic export to ONNX, Torchscript and CoreML. These 3 exports are currently working correctly when run in verified environments that meet all requirements.txt dependencies using the official models. 3rd party implementations that import exported models are not under our control and are beyond the scope of our support.

@lucasjinreal
Copy link

@glenn-jocher I solved the problem on cuda export, and it now also correct deploy to tensorrt.

@feolcn
Copy link

feolcn commented Sep 4, 2020

I met similar bug too. using openvino2020R4
Model Optimizer arguments:
Common parameters:
- Path to the Input Model: /data/aitest/yolov5/yolov5s.onnx
- Path for generated IR: /home/feol/intel/openvino_2020.4.287/deployment_tools/model_optimizer/.
- IR output name: yolov5s
- Log level: ERROR
- Batch: Not specified, inherited from the model
- Input layers: Not specified, inherited from the model
- Output layers: Not specified, inherited from the model
- Input shapes: [1,416,416,3]
- Mean values: Not specified
- Scale values: Not specified
- Scale factor: Not specified
- Precision of IR: FP32
- Enable fusing: True
- Enable grouped convolutions fusing: True
- Move mean values to preprocess section: False
- Reverse input channels: False
ONNX specific parameters:
Model Optimizer version:
[ ERROR ] Concat input shapes do not match
[ ERROR ] Shape is not defined for output 0 of "Concat_40".
[ ERROR ] Cannot infer shapes or values for node "Concat_40".
[ ERROR ] Not all output shapes were inferred or fully defined for node "Concat_40".
For more information please refer to Model Optimizer FAQ (https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html), question #40.
[ ERROR ]
[ ERROR ] It can happen due to bug in custom shape infer function <function concat_infer at 0x7f16ca187400>.
[ ERROR ] Or because the node inputs have incorrect values/shapes.
[ ERROR ] Or because input shapes are incorrect (embedded to the model or passed via --input_shape).
[ ERROR ] Run Model Optimizer with --log_level=DEBUG for more information.
[ ERROR ] Exception occurred during running replacer "REPLACEMENT_ID" (<class 'extensions.middle.PartialInfer.PartialInfer'>): Stopped shape/value propagation at "Concat_40" node.
For more information please refer to Model Optimizer FAQ (https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html), question #38.

@linhaoqi027
Copy link
Author

I met similar bug too. using openvino2020R4
Model Optimizer arguments:
Common parameters:

  • Path to the Input Model: /data/aitest/yolov5/yolov5s.onnx
  • Path for generated IR: /home/feol/intel/openvino_2020.4.287/deployment_tools/model_optimizer/.
  • IR output name: yolov5s
  • Log level: ERROR
  • Batch: Not specified, inherited from the model
  • Input layers: Not specified, inherited from the model
  • Output layers: Not specified, inherited from the model
  • Input shapes: [1,416,416,3]
  • Mean values: Not specified
  • Scale values: Not specified
  • Scale factor: Not specified
  • Precision of IR: FP32
  • Enable fusing: True
  • Enable grouped convolutions fusing: True
  • Move mean values to preprocess section: False
  • Reverse input channels: False
    ONNX specific parameters:
    Model Optimizer version:
    [ ERROR ] Concat input shapes do not match
    [ ERROR ] Shape is not defined for output 0 of "Concat_40".
    [ ERROR ] Cannot infer shapes or values for node "Concat_40".
    [ ERROR ] Not all output shapes were inferred or fully defined for node "Concat_40".
    For more information please refer to Model Optimizer FAQ (https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html), question C++ API #40.
    [ ERROR ]
    [ ERROR ] It can happen due to bug in custom shape infer function <function concat_infer at 0x7f16ca187400>.
    [ ERROR ] Or because the node inputs have incorrect values/shapes.
    [ ERROR ] Or because input shapes are incorrect (embedded to the model or passed via --input_shape).
    [ ERROR ] Run Model Optimizer with --log_level=DEBUG for more information.
    [ ERROR ] Exception occurred during running replacer "REPLACEMENT_ID" (<class 'extensions.middle.PartialInfer.PartialInfer'>): Stopped shape/value propagation at "Concat_40" node.
    For more information please refer to Model Optimizer FAQ (https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html), question Add results to sotabench #38.

Did your version is torch==1.5.1,torchvision==0.6.1 ?

@feolcn
Copy link

feolcn commented Sep 5, 2020

是的

@linhaoqi027
Copy link
Author

是的

I didn't meet this bug.May be it is cause by yolov5 version.I used yolov5_3

@FrzWang
Copy link

FrzWang commented Sep 9, 2020

I met similar bug too. using openvino2020R4+LeakyRelu()
Screenshot from 2020-09-03 22-57-22
Did your version is torch==1.5.1,torchvision==0.6.1 when convert torch to onnx?Higher version may cause problem(This problem will cause error when convert onnx to openvino)

Thanks for your reply. It works when i change torch==1.5.1,torchvision==0.6.1.
Nice work.

@makaveli10
Copy link

@glenn-jocher I solved the problem on cuda export, and it now also correct deploy to tensorrt.

@jinfagang what exactly did you do to convert v3.0 to tensorrt. Can you please throw some light on that?

@lucasjinreal
Copy link

@makaveli10 I am simply convert to onnx, not inference via tensorrt, however seems here is an out-of-box example: http://manaai.cn/aisolution_detail.html?id=5

@makaveli10
Copy link

@jinfagang so you didnt convert v3.0 to tensorrt. but able to convert to onnx?

@masouduut94
Copy link

masouduut94 commented Sep 11, 2020

@linhaoqi027
@jinfagang

I successfully converted the onnx model to ir model. while I was using your code on this link I noticed on this line that the outputs are far from expected.

I trained the model on 2 classes so I expected network values like [210, 85 , 266, 175, 0.78, 1], but I got [-0.58, .25, 1.14, 12, -0.53].

What I mean is that I expect the last result to be 0 or 1 (for two classes) and the 4th value to be between 0 to 1 for confidence.

How can I solve this issue?
Is it because I did not consider arguments scale_values and mean_values when using model_optimizer conversion on openvino?

Please explain in details.
Thank you

@linhaoqi027
Copy link
Author

linhaoqi027 commented Sep 16, 2020

@ linhaoqi027 @金发

我成功地将onnx模型转换为ir模型。当我在此链接上使用您的代码时,我在此行上注意到输出远远超出了预期。

我在2个班级上训练了模型,因此我希望[210, 85 , 266, 175, 0.78, 1]获得类似的网络价值,但是我得到了[-0.58, .25, 1.14, 12, -0.53]

我的意思是,我希望最后的结果是0或1(对于两个类),而第4个值的可信度在0到1之间。

我该如何解决这个问题?
是因为我没有考虑参数scale_values,还是mean_values在openvino上使用model_optimizer转换时?

请详细说明。
谢谢
This output is the model output,NMS is not applied.You should use the output after NMS. The NMS code is included in my project. You should observe the output after NMS in https://github.com/linhaoqi027/yolov5_openvino_sdk/blob/3cd625264171a6bc5a545932ca69191979dbc7a9/ji.py#L219

@github-actions
Copy link
Contributor

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the Stale label Oct 17, 2020
@JoshChristie
Copy link

The above steps worked for me when I revert back to commit c889267

@adityap27
Copy link

@JoshChristie @linhaoqi027
Yes these steps and given commit works for me.
But have you compared inference latency ?
It seems that there is no speed-up....or I am missing something.

@JoshChristie
Copy link

Yes, I see a noticeable decrease in inference time after converting the model to openvino for use on a CPU.

@adityap27
Copy link

adityap27 commented Oct 21, 2020

@JoshChristie
If there is no speed-up then there's not much use of this conversion to openvino.
Let me know any solution.

@JoshChristie
Copy link

JoshChristie commented Oct 21, 2020

@adityap27
To be clear, here is what I get:
CPU inference time = 30ms. NMS time added additional 15ms
CPU with openvino inference time = 17ms. NMS not attempted. Openvino version 2020.4.287. With ONNX simplifier.

CPU info i7-8750H CPU @ 2.20GHz

Model:

# parameters
nc: 1  # number of classes
depth_multiple: 0.2  # model depth multiple
width_multiple: 0.2  # layer channel multiple

~ 1.2mill parameters

@adityap27
Copy link

adityap27 commented Oct 21, 2020

@JoshChristie
Can you elaborate ? because 30ms to 17ms is a good speedup.
30ms is for pytorch model?
are these without NMS or with NMS?
which openvino version you are using ?
also which CPU model ?

I am getting same latency for both pytorch and openvino.
I need this conversion working with speedup asap.
It would be great if you elaborate about the above details I have asked for.

@JoshChristie
Copy link

@adityap27 Updated my previous comment.
Also, I used onnx simplifier but I don't know how much that changes things.

@adityap27
Copy link

adityap27 commented Oct 21, 2020

@JoshChristie
Thanks for the details.
I haven't tried with onnx-simplifier. I will try that.

  • Also are these default or have you changed ?
depth_multiple: 0.2  # model depth multiple
width_multiple: 0.2  # layer channel multiple
  • Also were the input shapes same for both inference and what 480 or 640 ?

  • Are you measuring this part ? in detect.py
    image

@linhaoqi027
Copy link
Author

@JoshChristie
如果没有加速,那么这种转换到openvino的用途就很少。
让我知道任何解决方案。

@linhaoqi027
Copy link
Author

@adityap27 Updated my previous comment.
Also, I used onnx simplifier but I don't know how much that changes things.

The nms is not included in openvino.So i think the main time waste is in nms. I convert output(numpy) to pytorch than use tensor to do nms.So I think that cause main time waste.

@adityap27
Copy link

adityap27 commented Oct 21, 2020

@linhaoqi027
Thanks for confirmation.
Can you tell me

  • what's the purpose to change torch1.5.1 and not to use 1.6 ?? (before export.py)
  • also have you noticed inference speedup(with/without nms) in openvino as compared to pytorch ?

@adityap27
Copy link

@JoshChristie
Thanks for the details.
I haven't tried with onnx-simplifier. I will try that.

* Also are these default or have you changed ?
depth_multiple: 0.2  # model depth multiple
width_multiple: 0.2  # layer channel multiple
* Also were the input shapes same for both inference and what 480 or 640 ?

* Are you measuring this part ? in detect.py
  ![image](https://user-images.githubusercontent.com/30194525/96680253-c79c6300-1392-11eb-9554-43049c72fd2b.png)

@JoshChristie can you help with these info ?

@linhaoqi027
Copy link
Author

@JoshChristie
感谢您提供详细信息。
我没有尝试过onnx-simplifier。我会尝试的。

* Also are these default or have you changed ?
depth_multiple: 0.2  # model depth multiple
width_multiple: 0.2  # layer channel multiple
* Also were the input shapes same for both inference and what 480 or 640 ?

* Are you measuring this part ? in detect.py
  ![image](https://user-images.githubusercontent.com/30194525/96680253-c79c6300-1392-11eb-9554-43049c72fd2b.png)

@JoshChristie您可以提供这些信息吗?
you can refer to https://github.com/linhaoqi027/yolov5_openvino_sdk.
if i use 1.6,it will occur an error (what's the purpose to change torch1.5.1 and not to use 1.6 ?? (before export.py))
And sorry i dont notice the inference speedup(with/without nms) in openvino as compared to pytorch

@manumenon
Copy link

having same size issue when trying to convert from onnx to ie

@linhaoqi027
Copy link
Author

尝试从onnx转换为ie时遇到相同的大小问题

having same size issue when trying to convert from onnx to ie

you can refer to https://github.com/linhaoqi027/yolov5_openvino_sdk.

@Belinda-great
Copy link

@adityap27
To be clear, here is what I get:
CPU inference time = 30ms. NMS time added additional 15ms
CPU with openvino inference time = 17ms. NMS not attempted. Openvino version 2020.4.287. With ONNX simplifier.

CPU info i7-8750H CPU @ 2.20GHz

Model:

# parameters
nc: 1  # number of classes
depth_multiple: 0.2  # model depth multiple
width_multiple: 0.2  # layer channel multiple

~ 1.2mill parameters

Hi, very hanppy for your information! I use yolov5s, v3, img size is 640*640, one thread, test openvino(2020R4) time is 1.28s, and pytorch one thread time is 1.2s, it's too slow.
platform: E5-2699 V3 @2.30GHz.
Is there anything wrong with my test? and what img size did you use for input?

@JoshChristie
Copy link

JoshChristie commented Nov 9, 2020

With one thread your results make sense to me. My image size was 288x512.

@mengqingmeng
Copy link

Have u guys tried:

python -m onnxsim a.onnx a_sim.onnx

to simplify your onnx model? This is really helpful to eliminate unsupported ops and simplify whole onnx model structure.

If you guys can simplify your onnx model pls let me know.

thanks a lot, your answer work for me

@glenn-jocher glenn-jocher linked a pull request Dec 23, 2021 that will close this issue
@glenn-jocher
Copy link
Member

@linhaoqi027 @violet17 @feolcn @JoshChristie @mengqingmeng @Belinda-great good news 😃! Your original issue may now be fixed ✅ in PR #6057. This PR adds native YOLOv5 OpenVINO export:

python export.py --weights yolov5s.pt --include openvino  # export to OpenVINO

image

To receive this update:

  • Gitgit pull from within your yolov5/ directory or git clone https://github.com/ultralytics/yolov5 again
  • PyTorch Hub – Force-reload model = torch.hub.load('ultralytics/yolov5', 'yolov5s', force_reload=True)
  • Notebooks – View updated notebooks Open In Colab Open In Kaggle
  • Dockersudo docker pull ultralytics/yolov5:latest to update your image Docker Pulls

Thank you for spotting this issue and informing us of the problem. Please let us know if this update resolves the issue for you, and feel free to inform us of any other issues you discover or feature requests that come to mind. Happy trainings with YOLOv5 🚀!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.