You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.
It's okay to run CPU version SSD exmaple, but there's something wrong with CUDA. The main cuda error log is: example/ssd/operator/multibox_detection.cu:199: Check failed: error == cudaSuccess (7 vs. 0) too many resources requested for launch
Environment info
Operating System: TegraX1 Ubuntu 16.04.2 LTS
Linux tegra-ubuntu 3.10.96-tegra
Compiler: gcc version 5.4.0 20160609 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.4)
Package used (Python/R/Scala/Julia):python
MXNet version:0.9.4
Or if installed from source: Yes
MXNet commit hash (git rev-parse HEAD):
If you are using python package, please provide
Python version and distribution: Python 2.7.12
Error Message:
Please paste the full error message, including stack trace.
(py-mxnet-SSD) yuanshuai@tegra-ubuntu:~/sdcard/code/mxnet_SSD_cuda_openblas/example/ssd$ python demo.py --epoch 0 --images ./data/demo/dog.jpg --thresh 0.5
[09:58:45] src/nnvm/legacy_json_util.cc:175: Loading symbol saved by previous version v0.8.0. Attempting to upgrade...
[09:58:53] /mnt/sdcard/yuanshuai/code/mxnet_SSD_cuda_openblas/dmlc-core/include/dmlc/./logging.h:300: [09:58:53] example/ssd/operator/multibox_detection.cu:199: Check failed: error == cudaSuccess (7 vs. 0) too many resources requested for launch
[09:58:53] /mnt/sdcard/yuanshuai/code/mxnet_SSD_cuda_openblas/dmlc-core/include/dmlc/./logging.h:300: [09:58:53] src/engine/./threaded_engine.h:336: [09:58:53] example/ssd/operator/multibox_detection.cu:199: Check failed: error == cudaSuccess (7 vs. 0) too many resources requested for launch
An fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.
terminate called after throwing an instance of 'dmlc::Error'
what(): [09:58:53] src/engine/./threaded_engine.h:336: [09:58:53] example/ssd/operator/multibox_detection.cu:199: Check failed: error == cudaSuccess (7 vs. 0) too many resources requested for launch
An fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.
I found there may be something wrong with mxnet-ssd code version.
I fixed it after I clone the original mxnet-ssd contributor's repo from his own code repo(https://github.com/zhreshold/mxnet-ssd) and then tried as its README says, finally it worked!
Before I fixed it, I referred this issue titled src/operator/multibox_detection.cu:198: Check failed: (error) == (cudaSuccess) too many resources requested for launch · Issue #13 · zhreshold/mxnet-ssd, however it didn't work for me. Because there is no make files in official ./mxnet/examples/ssd dir, I couldn't compile the modified cuda codes.
It's okay to run CPU version SSD exmaple, but there's something wrong with CUDA. The main cuda error log is: example/ssd/operator/multibox_detection.cu:199: Check failed: error == cudaSuccess (7 vs. 0) too many resources requested for launch
Environment info
Operating System: TegraX1 Ubuntu 16.04.2 LTS
Linux tegra-ubuntu 3.10.96-tegra
Compiler: gcc version 5.4.0 20160609 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.4)
Package used (Python/R/Scala/Julia):python
MXNet version:0.9.4
Or if installed from source: Yes
MXNet commit hash (
git rev-parse HEAD
):If you are using python package, please provide
Python version and distribution: Python 2.7.12
Error Message:
Please paste the full error message, including stack trace.
(py-mxnet-SSD) yuanshuai@tegra-ubuntu:~/sdcard/code/mxnet_SSD_cuda_openblas/example/ssd$ python demo.py --epoch 0 --images ./data/demo/dog.jpg --thresh 0.5
[09:58:45] src/nnvm/legacy_json_util.cc:175: Loading symbol saved by previous version v0.8.0. Attempting to upgrade...
[09:58:53] /mnt/sdcard/yuanshuai/code/mxnet_SSD_cuda_openblas/dmlc-core/include/dmlc/./logging.h:300: [09:58:53] example/ssd/operator/multibox_detection.cu:199: Check failed: error == cudaSuccess (7 vs. 0) too many resources requested for launch
Stack trace returned 7 entries:
[bt] (0) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x48) [0x7f737f2da0]
[bt] (1) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(ZN5mxnet2op19MultiBoxDetectionOpIN7mshadow3gpuEfE7ForwardERKNS_9OpContextERKSt6vectorINS_5TBlobESaIS9_EERKS8_INS_9OpReqTypeESaISE_EESD_SD+0x1b18) [0x7f74a34a40]
[bt] (2) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(+0xf2a048) [0x7f740f5048]
[bt] (3) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x88) [0x7f74095b68]
[bt] (4) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x88) [0x7f7409c6c0]
[bt] (5) /usr/lib/aarch64-linux-gnu/libstdc++.so.6(+0xb8280) [0x7f66e24280]
[bt] (6) /lib/aarch64-linux-gnu/libpthread.so.0(+0x6fc4) [0x7f7e02dfc4]
[09:58:53] /mnt/sdcard/yuanshuai/code/mxnet_SSD_cuda_openblas/dmlc-core/include/dmlc/./logging.h:300: [09:58:53] src/engine/./threaded_engine.h:336: [09:58:53] example/ssd/operator/multibox_detection.cu:199: Check failed: error == cudaSuccess (7 vs. 0) too many resources requested for launch
Stack trace returned 7 entries:
[bt] (0) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x48) [0x7f737f2da0]
[bt] (1) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(ZN5mxnet2op19MultiBoxDetectionOpIN7mshadow3gpuEfE7ForwardERKNS_9OpContextERKSt6vectorINS_5TBlobESaIS9_EERKS8_INS_9OpReqTypeESaISE_EESD_SD+0x1b18) [0x7f74a34a40]
[bt] (2) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(+0xf2a048) [0x7f740f5048]
[bt] (3) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x88) [0x7f74095b68]
[bt] (4) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x88) [0x7f7409c6c0]
[bt] (5) /usr/lib/aarch64-linux-gnu/libstdc++.so.6(+0xb8280) [0x7f66e24280]
[bt] (6) /lib/aarch64-linux-gnu/libpthread.so.0(+0x6fc4) [0x7f7e02dfc4]
An fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.
Stack trace returned 5 entries:
[bt] (0) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x48) [0x7f737f2da0]
[bt] (1) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x2d0) [0x7f74095db0]
[bt] (2) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x88) [0x7f7409c6c0]
[bt] (3) /usr/lib/aarch64-linux-gnu/libstdc++.so.6(+0xb8280) [0x7f66e24280]
[bt] (4) /lib/aarch64-linux-gnu/libpthread.so.0(+0x6fc4) [0x7f7e02dfc4]
terminate called after throwing an instance of 'dmlc::Error'
what(): [09:58:53] src/engine/./threaded_engine.h:336: [09:58:53] example/ssd/operator/multibox_detection.cu:199: Check failed: error == cudaSuccess (7 vs. 0) too many resources requested for launch
Stack trace returned 7 entries:
[bt] (0) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x48) [0x7f737f2da0]
[bt] (1) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(ZN5mxnet2op19MultiBoxDetectionOpIN7mshadow3gpuEfE7ForwardERKNS_9OpContextERKSt6vectorINS_5TBlobESaIS9_EERKS8_INS_9OpReqTypeESaISE_EESD_SD+0x1b18) [0x7f74a34a40]
[bt] (2) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(+0xf2a048) [0x7f740f5048]
[bt] (3) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x88) [0x7f74095b68]
[bt] (4) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x88) [0x7f7409c6c0]
[bt] (5) /usr/lib/aarch64-linux-gnu/libstdc++.so.6(+0xb8280) [0x7f66e24280]
[bt] (6) /lib/aarch64-linux-gnu/libpthread.so.0(+0x6fc4) [0x7f7e02dfc4]
An fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.
Stack trace returned 5 entries:
[bt] (0) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x48) [0x7f737f2da0]
[bt] (1) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x2d0) [0x7f74095db0]
[bt] (2) /mnt/sdcard/yuanshuai/virtualenvs/py-mxnet-SSD/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvvEZZN5mxnet6engine23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlvE_E9_M_invokeERKSt9_Any_data+0x88) [0x7f7409c6c0]
[bt] (3) /usr/lib/aarch64-linux-gnu/libstdc++.so.6(+0xb8280) [0x7f66e24280]
[bt] (4) /lib/aarch64-linux-gnu/libpthread.so.0(+0x6fc4) [0x7f7e02dfc4]
Aborted
The text was updated successfully, but these errors were encountered: