Errors that occur during training（训练出错）

[shenshaowei]

(sam3d) a@a-Super-Server:/media/a/DATA/ssw-baselines/SAM-Med3D$ python3 train.py
Loaded checkpoint from ckpt/sam_med3d.pth (epoch 0)
Epoch: 0/199
0%| | 0/150 [00:00<?, ?it/s]/home/a/miniconda3/envs/sam3d/lib/python3.9/site-packages/torch/nn/modules/conv.py:605: UserWarning: Plan failed with a cudnnException: CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed cudnn_status: CUDNN_STATUS_NOT_SUPPORTED (Triggered internally at ../aten/src/ATen/native/cudnn/Conv_v8.cpp:919.)
return F.conv3d(
1%|▌ | 1/150 [00:05<14:11, 5.72s/it]
Traceback (most recent call last):
File "/media/a/DATA/ssw-baselines/SAM-Med3D/train.py", line 520, in
main()
File "/media/a/DATA/ssw-baselines/SAM-Med3D/train.py", line 479, in main
trainer.train()
File "/media/a/DATA/ssw-baselines/SAM-Med3D/train.py", line 374, in train
epoch_loss, epoch_iou, epoch_dice, pred_list = self.train_epoch(epoch, num_clicks)
File "/media/a/DATA/ssw-baselines/SAM-Med3D/train.py", line 294, in train_epoch
for step, (image3D, gt3D) in enumerate(tbar):
File "/home/a/miniconda3/envs/sam3d/lib/python3.9/site-packages/tqdm/std.py", line 1181, in iter
for obj in iterable:
File "/home/a/miniconda3/envs/sam3d/lib/python3.9/site-packages/prefetch_generator/init.py", line 116, in next
raise next_item
File "/home/a/miniconda3/envs/sam3d/lib/python3.9/site-packages/prefetch_generator/init.py", line 98, in run
for item in self.generator: self.queue.put((True , item))
File "/home/a/miniconda3/envs/sam3d/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 631, in next
data = self._next_data()
File "/home/a/miniconda3/envs/sam3d/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1326, in _next_data
return self._process_data(data)
File "/home/a/miniconda3/envs/sam3d/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1372, in _process_data
data.reraise()
File "/home/a/miniconda3/envs/sam3d/lib/python3.9/site-packages/torch/_utils.py", line 705, in reraise
raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 1.
Original Traceback (most recent call last):
File "/home/a/miniconda3/envs/sam3d/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
data = fetcher.fetch(index) # type: ignore[possibly-undefined]
File "/home/a/miniconda3/envs/sam3d/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 54, in fetch
return self.collate_fn(data)
File "/home/a/miniconda3/envs/sam3d/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 316, in default_collate
return collate(batch, collate_fn_map=default_collate_fn_map)
File "/home/a/miniconda3/envs/sam3d/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 173, in collate
return [collate(samples, collate_fn_map=collate_fn_map) for samples in transposed] # Backwards compatibility.
File "/home/a/miniconda3/envs/sam3d/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 173, in
return [collate(samples, collate_fn_map=collate_fn_map) for samples in transposed] # Backwards compatibility.
File "/home/a/miniconda3/envs/sam3d/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 141, in collate
return collate_fn_map[elem_type](batch, collate_fn_map=collate_fn_map)
File "/home/a/miniconda3/envs/sam3d/lib/python3.9/site-packages/torch/utils/data/_utils/collate.py", line 213, in collate_tensor_fn
return torch.stack(batch, 0, out=out)
RuntimeError: torch.cat(): input types can't be cast to the desired output type Int

It's been stuck for a long time, is there a solution?（卡了很久了，有解决办法吗？）

[shenshaowei]

这是错误截图

[RRouhi]

I got the same error "RuntimeError: torch.cat(): input types can't be cast to the desired output type Int". Setting the --batch_size 1 solved the issue.

[tuan-ld]

I got the same error "RuntimeError: torch.cat(): input types can't be cast to the desired output type Int". Setting the --batch_size 1 solved the issue.

Thank you, it works!