Training speed become slower iteration by iteration

[philokey]

Hello,

When I training the model, the speed of per iteration slow down, eg. in the beginning, the speed is about 0.4s/iter, after 10000 iterations, the speed reduce to about 1s/iter. However, the time of tensorflow session
rpn_loss_cls_value, rpn_loss_box_value,loss_cls_value, loss_box_value, _ = sess.run([rpn_cross_entropy, rpn_loss_box, cross_entropy, loss_box, train_op], feed_dict=feed_dict)
does not increase.

What's more, the CPU time seems much more than the beginning, the usage of GPU is often 0%. Therefore, I suspect that there are something wrong in roi_data_layer which run in CPU.

I have check the code, but I can not find any bug. Has anyone meets this problem and how to solve this problem.

Thank you.

[philokey]

I find the reason.
In training processing, the speed of the following code will slow down (I still do not know why).

if iter >= cfg.TRAIN.STEPSIZE:
     sess.run(tf.assign(lr, cfg.TRAIN.LEARNING_RATE * cfg.TRAIN.GAMMA))
else:
    sess.run(tf.assign(lr, cfg.TRAIN.LEARNING_RATE))

Hence, set the learning rate out of the loop can avoid this problem.

[flowice]

@philokey Hi, I also find the problem. What do you mean by "set the learning rate out of the loop can avoid this problem"? I think it is weird. Have you found the reason?

[philokey]

@flowice when you set learning rate, it will add a new node in tensorflow's graph, therefore, if you set learning rate in every iteration, it will add many nodes in the graph and become very slow.

[flowice]

@philokey Wonderful ! I will have a try. Thank you.