You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In the ProMISe issue, it was mentioned that this project requires around 24GB of VRAM.
However, I still encounter out-of-memory issue with RTX 3090 (24GB)
while running the following command:
python src/train.py --data colon --data_dir "dataset/Task10_Colon" --save_name "my_train" --multiple_outputs --dynamic --use_box --refine
[11:35:04.209] Namespace(data='colon', save_dir='./implementation/colon/my_train', data_dir='dataset/Task10_Colon', num_workers=2, split='train', use_small_dataset=False, model_type='vit_b_ori', lr=4e-05, lr_scheduler='linear', warm_up=False, device='cuda:0', max_epoch=200, image_size=128, batch_size=1, checkpoint='best', checkpoint_sam='./checkpoint_sam/sam_vit_b_01ec64.pth', num_classes=2, tolerance=5, boundary_kernel_size=5, use_pretrain=False, pretrain_path='', resume=False, resume_best=False, ddp=False, gpu_ids=[0, 1], accumulation_steps=20, iter_nums=11, num_clicks=50, num_clicks_validation=10, use_box=True, dynamic_box=False, use_scribble=False, num_multiple_outputs=3, multiple_outputs=True, refine=True, no_detach=False, refine_test=False, dynamic=True, efficient_scribble=False, use_sam3d_turbo=False, save_predictions=False, save_csv=False, save_test_dir='./', save_name='my_train')
2024-08-15 11:35:04,209 - Namespace(data='colon', save_dir='./implementation/colon/my_train', data_dir='dataset/Task10_Colon', num_workers=2, split='train', use_small_dataset=False, model_type='vit_b_ori', lr=4e-05, lr_scheduler='linear', warm_up=False, device='cuda:0', max_epoch=200, image_size=128, batch_size=1, checkpoint='best', checkpoint_sam='./checkpoint_sam/sam_vit_b_01ec64.pth', num_classes=2, tolerance=5, boundary_kernel_size=5, use_pretrain=False, pretrain_path='', resume=False, resume_best=False, ddp=False, gpu_ids=[0, 1], accumulation_steps=20, iter_nums=11, num_clicks=50, num_clicks_validation=10, use_box=True, dynamic_box=False, use_scribble=False, num_multiple_outputs=3, multiple_outputs=True, refine=True, no_detach=False, refine_test=False, dynamic=True, efficient_scribble=False, use_sam3d_turbo=False, save_predictions=False, save_csv=False, save_test_dir='./', save_name='my_train')
Unet_encoder features: (32, 32, 64, 128, 384, 32).
Unet_decoder features: (32, 32, 64, 128, 384, 32).
dataloaders are created, models are loaded, and others are set, spent 3.8 for rank -1
num_clicks 50 points_length: 69189 dynamic_size: 13
First batch: fn: 1.0000, fp: 0.0000, label 0: tensor(0), label 1: tensor(13)
--- ===================================== ---
--- above before model, below after model ---
--- ===================================== ---
dice before refine 0.07599078863859177 and after 0.037667643278837204
num_clicks 50 points_length: 356190 dynamic_size: 11
First batch: fn: 0.8992, fp: 4.2488, label 0: tensor(11), label 1: tensor(0)
--- ===================================== ---
......
RuntimeError: CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 23.69 GiB total capacity; 21.36 GiB already allocated; 42.62 MiB free; 21.59 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
How can I resolve this problem?
The text was updated successfully, but these errors were encountered:
Hi, Sorry for replying so late.
In the ProMISe issue, it was mentioned that this project requires around 24GB of VRAM.
However, I still encounter out-of-memory issue with RTX 3090 (24GB)
while running the following command:
How can I resolve this problem?
The text was updated successfully, but these errors were encountered: