Layers missing call to cudaStreamSynchronize?

[seanbell]

Hi, I am trying to port my code to version 0.16, and trying to understand cudaStreamSynchronize. When is it needed? The different layers seem to inconsistently use it.

For example, it is included here:
https://github.com/NVIDIA/caffe/blob/caffe-0.16/src/caffe/layers/relu_layer.cu#L54
https://github.com/NVIDIA/caffe/blob/caffe-0.16/src/caffe/layers/bias_layer.cu#L27
https://github.com/NVIDIA/caffe/blob/caffe-0.16/src/caffe/layers/bnll_layer.cu#L26
...

But not here:
https://github.com/NVIDIA/caffe/blob/caffe-0.16/src/caffe/layers/scale_layer.cu#L47
https://github.com/NVIDIA/caffe/blob/caffe-0.16/src/caffe/layers/slice_layer.cu#L40
https://github.com/NVIDIA/caffe/blob/caffe-0.16/src/caffe/layers/threshold_layer.cu#L22
https://github.com/NVIDIA/caffe/blob/caffe-0.16/src/caffe/layers/tile_layer.cu#L28
https://github.com/NVIDIA/caffe/blob/caffe-0.16/src/caffe/layers/batch_reindex_layer.cu#L30
https://github.com/NVIDIA/caffe/blob/caffe-0.16/src/caffe/layers/tanh_layer.cu#L24
https://github.com/NVIDIA/caffe/blob/caffe-0.16/src/caffe/layers/crop_layer.cu#L70
...
(Some are potentially also missing the kernel check)

These are incomplete lists, more files may be missing it.

[seanbell]

I think it's also missing from these kernel calls in math_functions: https://github.com/NVIDIA/caffe/blob/caffe-0.16/include/caffe/util/math_functions.hpp#L333-L387 (unless I'm misunderstanding something)

[drnikolaev]

Hi @seanbell Thank you. It's actually needed everywhere to make sure that compute on that stream is done because when you specify a non-zero stream for a kernel, then that kernel runs asynchronously. There are exclusions though:

• When you have chain of calls on the same stream, they do not overlap. Just put cSS after the last one.
• You run 2 kernels on 2 separate streams. If GPU can handle this load, it runs them simultaneously. Then you call cSS one time per each stream to make sure that compute is done.

So, those spots were missed and I'll fix them asap. But majority of them is used in chains of other calls thus they are properly synchronized (in other words, it's not that bad :)

[drnikolaev]

Fixed.