modulated_deformable_im2col_cuda: no kernel image is available for execution on the device

[MohammadAminDHM]

Hi @dongrunmin , I get this error when I tried to train the network :

error in modulated_deformable_im2col_cuda: no kernel image is available for execution on the device
Traceback (most recent call last):
  File "train.py", line 256, in <module>
    main()
  File "train.py", line 182, in main
    model.optimize_parameters(current_step)
  File "/content/drive/MyDrive/Projects/RRSGAN/RRSGAN/codes/models/RRSGAN_model.py", line 249, in optimize_parameters
    self.fake_H = self.netG(self.var_LQ, self.var_LQ_UX4, self.var_Ref, self.var_Ref_DUX4)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/parallel/data_parallel.py", line 141, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/content/drive/MyDrive/Projects/RRSGAN/RRSGAN/codes/models/archs/RRSNet_arch.py", line 61, in forward
    Ref_conv1, Ref_conv2, Ref_conv3 = self.gpcd_align(Ref_use_fea_l,Ref_fea_l,LR_fea_l)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/content/drive/MyDrive/Projects/RRSGAN/RRSGAN/codes/models/archs/RRSNet_arch.py", line 119, in forward
    L3_fea_output = self.lrelu(self.L3_fea_conv(L3_fea))
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py", line 320, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR

can you help me to debug that??

[dongrunmin]

It seems that your Pytorch version is inconsistent with CUDA version. Please first confirm your Pytorch can successfully use the GPU. You can use the following instructions:
import torch
torch.cuda.is_available()

[MohammadAminDHM]

My Pytorch version is 11.1 and I use colab to training model, also I use 'img' mode to training model. in your opinion, the reason for this error could be due to the use of the colab?? @dongrunmin

[dongrunmin]

I'm not sure. This could help pytorch/pytorch#16831 (comment)

[MohammadAminDHM]

Thanks, I consider it