bert->onnx ->caffe2 weird error

[maeotaku]

So really not sure if i should post this here but im having this problem with the pretrained bert for seq classification in particular when i try to consume the ONNX version of the model with Caffe2, I get this output:

File "/usr/local/lib/python3.6/dist-packages/caffe2/python/onnx/workspace.py", line 63, in f
return getattr(workspace, attr)(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/caffe2/python/workspace.py", line 250, in RunNet
StringifyNetName(name), num_iter, allow_fail,
File "/usr/local/lib/python3.6/dist-packages/caffe2/python/workspace.py", line 211, in CallWithExceptionIntercept
return func(args, kwargs)
RuntimeError: [enforce fail at pow_op.h:100] A.sizes() == B.sizes(). [4, 512, 768] vs []. Dimension mismatch - did you forget to set broadcast=1?
Error from operator:
input: "222" input: "223" output: "224" name: "" type: "Pow" device_option { device_type: 1 device_id: 3 }frame #0: std::function<std::string ()>::operator()() const + 0x11 (0x7f9c8cdaf441 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so)
frame #1: c10::ThrowEnforceNotMet(char const, int, char const, std::string const&, void const) + 0x49 (0x7f9c8cdaf259 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so)
frame #2: + 0x2b63861 (0x7f9c44eed861 in /usr/local/lib/python3.6/dist-packages/torch/lib/libcaffe2_gpu.so)
frame #3: + 0x15a3555 (0x7f9c4392d555 in /usr/local/lib/python3.6/dist-packages/torch/lib/libcaffe2_gpu.so)
frame #4: caffe2::SimpleNet::Run() + 0x161 (0x7f9c396ac101 in /usr/local/lib/python3.6/dist-packages/torch/lib/libcaffe2.so)
frame #5: caffe2::Workspace::RunNet(std::string const&) + 0x3a (0x7f9c396e35aa in /usr/local/lib/python3.6/dist-packages/torch/lib/libcaffe2.so)
frame #6: + 0x4e38a (0x7f9bbe6fd38a in /usr/local/lib/python3.6/dist-packages/caffe2/python/caffe2_pybind11_state_gpu.cpython-36m-x86_64-linux-gnu.so)
frame #7: + 0x9368e (0x7f9bbe74268e in /usr/local/lib/python3.6/dist-packages/caffe2/python/caffe2_pybind11_state_gpu.cpython-36m-x86_64-linux-gnu.so)
frame #8: PyCFunction_Call + 0xf9 (0x4aeb29 in /usr/bin/python3)
frame #9: _PyEval_EvalFrameDefault + 0x7e42 (0x54d092 in /usr/bin/python3)
frame #10: /usr/bin/python3() [0x543f21]
frame #11: /usr/bin/python3() [0x54421f]
frame #12: _PyEval_EvalFrameDefault + 0xc5b (0x545eab in /usr/bin/python3)
frame #13: /usr/bin/python3() [0x543f21]
frame #14: PyEval_EvalCodeEx + 0x6d (0x544cfd in /usr/bin/python3)
frame #15: /usr/bin/python3() [0x485857]
frame #16: PyObject_Call + 0x60 (0x4557a0 in /usr/bin/python3)
frame #17: _PyEval_EvalFrameDefault + 0x19e8 (0x546c38 in /usr/bin/python3)
frame #18: /usr/bin/python3() [0x543f21]
frame #19: /usr/bin/python3() [0x54421f]
frame #20: _PyEval_EvalFrameDefault + 0xc5b (0x545eab in /usr/bin/python3)
frame #21: /usr/bin/python3() [0x543f21]
frame #22: /usr/bin/python3() [0x54421f]
frame #23: _PyEval_EvalFrameDefault + 0xc5b (0x545eab in /usr/bin/python3)
frame #24: /usr/bin/python3() [0x5432b1]
frame #25: /usr/bin/python3() [0x544447]
frame #26: _PyEval_EvalFrameDefault + 0xc5b (0x545eab in /usr/bin/python3)
frame #27: /usr/bin/python3() [0x5432b1]
frame #28: /usr/bin/python3() [0x544447]
frame #29: _PyEval_EvalFrameDefault + 0xc5b (0x545eab in /usr/bin/python3)
frame #30: /usr/bin/python3() [0x543f21]
frame #31: PyEval_EvalCodeEx + 0x6d (0x544cfd in /usr/bin/python3)
frame #32: /usr/bin/python3() [0x485857]
frame #33: PyObject_Call + 0x60 (0x4557a0 in /usr/bin/python3)
frame #34: _PyEval_EvalFrameDefault + 0x19e8 (0x546c38 in /usr/bin/python3)
frame #35: /usr/bin/python3() [0x543f21]
frame #36: PyEval_EvalCodeEx + 0x6d (0x544cfd in /usr/bin/python3)
frame #37: /usr/bin/python3() [0x485857]
frame #38: PyObject_Call + 0x60 (0x4557a0 in /usr/bin/python3)
frame #39: _PyEval_EvalFrameDefault + 0x19e8 (0x546c38 in /usr/bin/python3)
frame #40: /usr/bin/python3() [0x5432b1]
frame #41: /usr/bin/python3() [0x544447]
frame #42: _PyEval_EvalFrameDefault + 0xc5b (0x545eab in /usr/bin/python3)
frame #43: /usr/bin/python3() [0x5432b1]
frame #44: /usr/bin/python3() [0x544447]
frame #45: _PyEval_EvalFrameDefault + 0xc5b (0x545eab in /usr/bin/python3)
frame #46: /usr/bin/python3() [0x5432b1]
frame #47: _PyFunction_FastCallDict + 0x236 (0x54d8c6 in /usr/bin/python3)
frame #48: _PyObject_FastCallDict + 0x1ef (0x455acf in /usr/bin/python3)
frame #49: _PyObject_Call_Prepend + 0xcb (0x455bcb in /usr/bin/python3)
frame #50: PyObject_Call + 0x60 (0x4557a0 in /usr/bin/python3)
frame #51: /usr/bin/python3() [0x4c9d13]
frame #52: _PyObject_FastCallDict + 0xa2 (0x455982 in /usr/bin/python3)
frame #53: /usr/bin/python3() [0x544075]
frame #54: _PyEval_EvalFrameDefault + 0xc5b (0x545eab in /usr/bin/python3)
frame #55: /usr/bin/python3() [0x5432b1]
frame #56: /usr/bin/python3() [0x544447]
frame #57: _PyEval_EvalFrameDefault + 0xc5b (0x545eab in /usr/bin/python3)
frame #58: /usr/bin/python3() [0x5432b1]
frame #59: /usr/bin/python3() [0x544447]
frame #60: _PyEval_EvalFrameDefault + 0xc5b (0x545eab in /usr/bin/python3)
frame #61: /usr/bin/python3() [0x5432b1]
frame #62: _PyFunction_FastCallDict + 0x236 (0x54d8c6 in /usr/bin/python3)
frame #63: _PyObject_FastCallDict + 0x1ef (0x455acf in /usr/bin/python3)

Does any of you know if the pretrained model is using something not supported by Caffe2?
I have also tried with several tensor shapes ( like (1, 512), (1, 128), (1,512, 786) in both long anf float with no luck. Also i used (4, 512), (4,128), (4,512,768) just in case since my input when i exported the ONNX file used some 4 samples.

Any pointers would be highly appreciated :)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[kkaehler]

@maeotaku were you ever able to figure this out? I'd be curious to see what numbers you were seeing when running in caffe2.

If you didn't figure this out, seems similar this pytorch issue