The process of main_train.py never ends

[syoukera]

I have built a docker container based on the Dockerfile supplied in here. After confirmation of the output of demo.py, I tried to test the training processes.

docker exec serene_joliot python main_train.py

This is the log file of this process.

append flipped images to roidb
loading annotations into memory...
Done (t=13.01s)
creating index...
index created!
num_images 40504
COCO_val2014 gt roidb loaded from ./data/cache/COCO_val2014_gt_roidb.pkl
appending ground truth annotations
Reading cached proposals after ***NMS**** from data/proposals/COCO_val2014_rpn_after_nms.pkl
Done!
append flipped images to roidb
filtered 2138 roidb entries: 246574 -> 244436
add bounding box regression targets
bbox target means:
[[0. 0. 0. 0.]
 [0. 0. 0. 0.]]
[0. 0. 0. 0.]
bbox target stdevs:
[[0.1 0.1 0.2 0.2]
 [0.1 0.1 0.2 0.2]]
[0.1 0.1 0.2 0.2]
Creating Iterator with 244436 Images

I spent almost two days running this command. However, the output of this process was stopped at final line of the massage.

Does anyone have any idea to understand this state?

---

Machine spec
CPU: i7-8700K
GPU: GeForce GTX 1080