How much memory does it need to fine-tune full-parameter flux?

[Adoni]

I try to fine tune full-parameter of flux on H100 (80G cuda memory), with batch size=1 and resolution = 512.
But it crashed with CUDA Out of Memory. So I'm wondering how much memory it need.

I'm using the dream booth code and didn't change anything of it.

[bmaxdk]

Even with your 80GB of memory on an H100, you will loos cuda memory. The model size, resolution, and the memory needed for training like storing activations and gradients are too high for the available memory. Could you try with turn on gradient chckpt and mixed precision?
Also lower the image resolution to see whether it give issue

[Adoni]

Thanks. Actually, I've already turned on the gradient chckpt and used mixed precision to "bf16". I'm using resolution as 512. I will try smaller resolution such as 256 or even 128.

Here is more details when I turned on the gradient chckpt and used mixed precision to "bf16":

• Before training (before "for epoch" line), it used 54GB cuda memory.
• When I start training, it crashed because of cuda out of memory in forward stage

[Adoni]

I tried to reduce batch size to 128 and it still OOM