Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

opencv error during training process #1924

Closed
YoungSharp opened this issue Dec 20, 2017 · 1 comment
Closed

opencv error during training process #1924

YoungSharp opened this issue Dec 20, 2017 · 1 comment

Comments

@YoungSharp
Copy link

YoungSharp commented Dec 20, 2017

I1220 07:38:26.009254 19032 solver.cpp:362] Iteration 3860, Testing net (#0)
I1220 07:38:26.009270 19032 net.cpp:723] Ignoring source layer train_data
I1220 07:38:26.009274 19032 net.cpp:723] Ignoring source layer train_label
I1220 07:38:26.009274 19032 net.cpp:723] Ignoring source layer train_transform
I1220 07:40:26.499217 19032 solver.cpp:429] Test net output #0: loss_bbox = 1.63662 (* 2 = 3.27324 loss)
I1220 07:40:26.499321 19032 solver.cpp:429] Test net output #1: loss_coverage = 10.7671 (* 1 = 10.7671 loss)
I1220 07:40:26.499325 19032 solver.cpp:429] Test net output #2: mAP = 0
I1220 07:40:26.499328 19032 solver.cpp:429] Test net output #3: precision = 0
I1220 07:40:26.499331 19032 solver.cpp:429] Test net output #4: recall = 0
I1220 07:41:21.482302 19032 solver.cpp:242] Iteration 3888 (0.226819 iter/s, 211.623s/48 iter), loss = 13.1519
I1220 07:41:21.482405 19032 solver.cpp:261] Train net output #0: loss_bbox = 1.5995 (* 2 = 3.199 loss)
I1220 07:41:21.482411 19032 solver.cpp:261] Train net output #1: loss_coverage = 13.5516 (* 1 = 13.5516 loss)
I1220 07:41:21.482417 19032 sgd_solver.cpp:106] Iteration 3888, lr = 7.54746e-06
I1220 07:42:52.471341 19032 solver.cpp:242] Iteration 3936 (0.527531 iter/s, 90.99s/48 iter), loss = 24.961
I1220 07:42:52.471431 19032 solver.cpp:261] Train net output #0: loss_bbox = 0.981416 (* 2 = 1.96283 loss)
I1220 07:42:52.471437 19032 solver.cpp:261] Train net output #1: loss_coverage = 0.995832 (* 1 = 0.995832 loss)
I1220 07:42:52.471444 19032 sgd_solver.cpp:106] Iteration 3936, lr = 7.50547e-06
I1220 07:44:23.474804 19032 solver.cpp:242] Iteration 3984 (0.52745 iter/s, 91.0039s/48 iter), loss = 6.86588
I1220 07:44:23.474898 19032 solver.cpp:261] Train net output #0: loss_bbox = 1.11368 (* 2 = 2.22736 loss)
I1220 07:44:23.474903 19032 solver.cpp:261] Train net output #1: loss_coverage = 3.99163 (* 1 = 3.99163 loss)
I1220 07:44:23.474910 19032 sgd_solver.cpp:106] Iteration 3984, lr = 7.46417e-06
I1220 07:45:54.492636 19032 solver.cpp:242] Iteration 4032 (0.527369 iter/s, 91.0179s/48 iter), loss = 18.8171
I1220 07:45:54.492748 19032 solver.cpp:261] Train net output #0: loss_bbox = 1.66934 (* 2 = 3.33869 loss)
I1220 07:45:54.492753 19032 solver.cpp:261] Train net output #1: loss_coverage = 9.52689 (* 1 = 9.52689 loss)
I1220 07:45:54.492760 19032 sgd_solver.cpp:106] Iteration 4032, lr = 7.42355e-06
I1220 07:47:25.475150 19032 solver.cpp:242] Iteration 4080 (0.527576 iter/s, 90.9822s/48 iter), loss = 14.7126
I1220 07:47:25.475260 19032 solver.cpp:261] Train net output #0: loss_bbox = 4.48436 (* 2 = 8.96873 loss)
I1220 07:47:25.475265 19032 solver.cpp:261] Train net output #1: loss_coverage = 11.6653 (* 1 = 11.6653 loss)
I1220 07:47:25.475271 19032 sgd_solver.cpp:106] Iteration 4080, lr = 7.38358e-06
I1220 07:48:56.459316 19032 solver.cpp:242] Iteration 4128 (0.527568 iter/s, 90.9836s/48 iter), loss = 22.178
I1220 07:48:56.459406 19032 solver.cpp:261] Train net output #0: loss_bbox = 0.91269 (* 2 = 1.82538 loss)
I1220 07:48:56.459411 19032 solver.cpp:261] Train net output #1: loss_coverage = 6.5104 (* 1 = 6.5104 loss)
I1220 07:48:56.459419 19032 sgd_solver.cpp:106] Iteration 4128, lr = 7.34426e-06
I1220 07:50:27.478549 19032 solver.cpp:242] Iteration 4176 (0.527366 iter/s, 91.0184s/48 iter), loss = 10.1156
I1220 07:50:27.478693 19032 solver.cpp:261] Train net output #0: loss_bbox = 6.9837 (* 2 = 13.9674 loss)
I1220 07:50:27.478699 19032 solver.cpp:261] Train net output #1: loss_coverage = 15.8192 (* 1 = 15.8192 loss)
I1220 07:50:27.478705 19032 sgd_solver.cpp:106] Iteration 4176, lr = 7.30555e-06
I1220 07:51:58.494304 19032 solver.cpp:242] Iteration 4224 (0.527388 iter/s, 91.0147s/48 iter), loss = 13.4028
I1220 07:51:58.494428 19032 solver.cpp:261] Train net output #0: loss_bbox = 2.07173 (* 2 = 4.14347 loss)
I1220 07:51:58.494433 19032 solver.cpp:261] Train net output #1: loss_coverage = 6.74864 (* 1 = 6.74864 loss)
I1220 07:51:58.494441 19032 sgd_solver.cpp:106] Iteration 4224, lr = 7.26745e-06
I1220 07:52:38.301617 19032 solver.cpp:362] Iteration 4246, Testing net (#0)
I1220 07:52:38.301697 19032 net.cpp:723] Ignoring source layer train_data
I1220 07:52:38.301700 19032 net.cpp:723] Ignoring source layer train_label
I1220 07:52:38.301702 19032 net.cpp:723] Ignoring source layer train_transform
OpenCV Error: Null pointer (create() called for the missing output array) in create, file /tmp/binarydeb/ros-kinetic-opencv3-3.2.0/modules/core/src/matrix.cpp, line 2492
OpenCV Error: Null pointer (create() called for the missing output array) in create, file /tmp/binarydeb/ros-kinetic-opencv3-3.2.0/modules/core/src/matrix.cpp, line 2492
Traceback (most recent call last):
File "/home/young/SoftWare/caffe/python/caffe/layers/detectnet/clustering.py", line 133, in forward
bbox = cluster(self, data0, bottom[1].data)
File "/home/young/SoftWare/caffe/python/caffe/layers/detectnet/clustering.py", line 227, in cluster
boxes_cur_image = vote_boxes(propose_boxes, propose_cvgs, mask, self)
File "/home/young/SoftWare/caffe/python/caffe/layers/detectnet/clustering.py", line 193, in vote_boxes
self.gridbox_rect_eps)
cv2.error: /tmp/binarydeb/ros-kinetic-opencv3-3.2.0/modules/core/src/matrix.cpp:2492: error: (-27) create() called for the missing output array in function create

And I got this error in terminal " Train Caffe Model task failed with error code 1
".

I had tried three kind of detection mission. After I trained for a while, it gave me this error. Is anyone getting this error?

Thanks.

@YoungSharp YoungSharp changed the title opencv error when trained for while opencv error during training process Dec 20, 2017
@YoungSharp
Copy link
Author

I solved this by changing the opencv version to libopencv4tegra and libopencv4tegra-dev libopencv4tegra-repo libopencv4tegra-python.
So I guess it's the opencv environment causes the fault.
After this change my training goes well, and the mAP value goes right after several iteration.
Just in case, some one else would fall into this trouble.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant