getting this error torch.cuda.OutOfMemoryError: CUDA out of memory while running diffusers/examples/dreambooth/train_dreambooth.py

[akashlinux10may]

trying following command and getting the error

!accelerate launch diffusers/examples/dreambooth/train_dreambooth.py
--pretrained_model_name_or_path=CompVis/stable-diffusion-v1-4
--instance_data_dir=diffusers/Dog
--output_dir=diffusers/Dog_SD
--instance_prompt="a photo of dog

"
--resolution=512
--train_batch_size=1
--gradient_accumulation_steps=1
--learning_rate=5e-6
--lr_scheduler="constant"
--lr_warmup_steps=0
--max_train_steps=400
--push_to_hub

return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 114.00 MiB (GPU 0; 9.50 GiB total capacity; 1.13 GiB already allocated; 24.94 MiB free; 1.17 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[asomoza]

can you try adding the --gradient_checkpointing argument and test it?

[asomoza]

probably I did it before the release of that version, just replace this:

!pip -q install diffusers

with this:

!pip install git+https://github.com/huggingface/diffusers.git

[akashlinux10may]

now everything is sorted as per your instructions, but here its giving new error, ValueError: Could not find the operator torchvision::nms. Please make sure you have already registered the operator and (if registered from C++) loaded it via torch.ops.load_library. is there any way you and me can connect virtually?

[akashlinux10may]

see these warning when I am running the 1st cell of your colab
This behaviour is the source of the following dependency conflicts.
torchaudio 2.1.0+cu121 requires torch==2.1.0, but you have torch 2.2.0+cu121 which is incompatible.
torchdata 0.7.0 requires torch==2.1.0, but you have torch 2.2.0+cu121 which is incompatible.
torchtext 0.16.0 requires torch==2.1.0, but you have torch 2.2.0+cu121 which is incompatible.

[akashlinux10may]

now I have restarted, cleared my cache, it seems its working fine. training got started.now as the training started, as I have given my own images folder with 5 images, folder name lap, its giving me this error, IsADirectoryError: [Errno 21] Is a directory: '/content/sample_data/lap/.ipynb_checkpoints'

[asomoza]

colab makes those hidden directories automatically when you create new ones, just make sure that where you put your images doesn't have anything else on it.

[asomoza]

sorry, I'm actually working right now and try to help when I have time, can't spend that much time on this.

You're having conflicts between the installations of torch and torchvision, try doing a restart session and run all

if that doesn't work, then something changed in colab between when I did it and now, sadly I can't use colab anymore since I used it for another answer and it doesn't let me run more sessions. When I can connect to it to again I can help more if no else hasn't done it.

[akashlinux10may]

I understand, thanks for your help, You already helped me a lot. Much appreciated your time.