Bug in newly added multi-gpu code

[philipp-fischer]

Trying out simple networks with multi-gpu segfaults because they don't have learnable parameters:

In parallel.c these lines are critical:

CUDA_CHECK(cudaMalloc(&data_, size_ * sizeof(Dtype)));
[...]
CUDA_CHECK(cudaMalloc(&diff_, size_ * sizeof(Dtype)));
[...]
CUDA_CHECK(cudaMalloc(&parent_grads_, size_ * sizeof(Dtype)));

If the net does not have learnable parameters, size_ will be 0 and cudaMalloc will return null pointers.

I currently work around this by adding +1 byte to the allocated size, but there should be a better fix.

[ronghanghu]

Right now, multi-gpu is only used for training, so maybe #2903 didn't expect one to do multi-gpu training without learnable parameters.

Anyway, this problem should be address soon.

[thatguymike]

More fundamentally, what does it mean to train without learnable parameters?

[ronghanghu]

More fundamentally, what does it mean to train without learnable parameters?

Indeed, it doesn't make sense to train without learnable parameters.

[philipp-fischer]

It can make a lot of sense.
Say I want to debug my data layer for multi-gpu usage, and just write the outputs to a file with a python layer..

Anyway, a segfault should never happen just because the network architecture doesn't make sense.

[ronghanghu]

A simple workaround provided in #2924. Debates can follow in new issues on whether allowing training without learnable parameter.