How much VRAM is needed for this?

[mjohanning99]

I was trying to run the sampling_ex.py, but no matter how low I set the num_candidates value (even if it's set to one or two), it always tells me that it has run out of memory. I am using an NVIDIA Quadro M5000 with 8 GB of VRAM.

[chiheonk]

In case of the released 1.3B model, total >5GB memory (1.3B * 4 bytes/per parameter) is required for placing the model on the GPU. Hence, I suspect that the model itself is already consuming the most of your GPU's VRAM (8GB), and there is not enough room to conduct an inference even with batch size 1.

One possible solution could be to cast your model to half-precision (float16) as

model.to(device=device, dtype=torch.float16)

in place of https://github.com/kakaobrain/minDALL-E/blob/0262c0281f9b34d43a751af6d593312416c85446/examples/sampling_ex.py#L36

Let me know if this works!

[mjohanning99]

Hello and thank you for your response! I have added the line you suggested, but I got the following error when running the sample script:

/home/mjo/.cache/minDALL-E/1.3B/tokenizer successfully restored..
/home/mjo/.cache/minDALL-E/1.3B/stage1_last.ckpt successfully restored..
/home/mjo/.cache/minDALL-E/1.3B/stage2_last.ckpt succesfully restored..
Traceback (most recent call last):
  File "/home/mjo/AI_Stuff/mindDALL-E/minDALL-E/sampling_ex.py", line 36, in <module>
    model.to(device=device, dtype=torch.float16)
NameError: name 'torch' is not defined

I had to manually import the torch module at the top of the file for it to run the program; the resulting images, however, are now all black. The image below was creating using the standard prompt in the sampling_ex.py script with a num_candidate value of 15.