Prompt-Tuning for text-to-image diffusion models

[AHHHZ975]

Hi, I have been looking for a simple example/script that shows how I can use the prompt-tuning technique to fine-tune the text encoder of a stable diffusion model. But I could not find any. Could you please introduce me if there is already one? If there is no implementation, I would appreciate any help/available resources for fine-tuning the text encoder with or without the PEFT library. Thanks!

[asomoza]

Hi, what is the prompt-tuning technique or what do you mean by it?

Also you need to specify the model arch you want to use, there's a whole world of difference between the Clip model that older models use and the T5 that the new ones use.

If you mean the CLIP models, you can learn how to fine tune it in this repo: https://github.com/zer0int/CLIP-fine-tune

[asomoza]

I wonder why the model won't learn anything. Because my intuition is that by freezing the unet model, we are seeking the domain of those prompts that yield the best output images ("best" in terms of the objective during the training) without changing the unet and without forgetting previous tasks. So, in this case, the fine-tuning process should guide us to a specific domain of text prompts that is more aligned with the fine-tuning objective. Am I right? Can you also please explain why you think freezing the unet won't work?

That's because I don't really know what are you training for, for example what task are we referring here.

When you say the fine-tuning process should guide us to a specific domain of text prompts that is more aligned with the fine-tuning objective, there won't be any problem if the fine-tuning objective is something the unet knows, for example generating images in a domain it was trained but if it's something the unet still doesn't know e.g.: medical pathologies (x-rays) or instruct pix to pix it won't work.

These are my assumptions though because I haven't trained something like this and haven't seen someone else do it.

do you think it is possible to replace the CLIP in the SD pipeline with the T5 which is already supported in the Peft library

It's possible, there was a project called ELLA that did this, they released the code and weights for SD 1.5. Maybe that will work for what you need.

[asomoza]

Am I right? So, can we conclude that only learning the text embedding space while keeping the unet frozen can give us some results?

Yes, but in my experience textual inversions only work with concepts the model already knows, it doesn't make it learn new concepts and that's why they aren't as popular as LoRAs. Textual Inversion these days are mostly used to replace negative prompts or to guide the model to, for example, generate better hands (which in my experience never worked).

If you want to do something similar, in theory, it should also work.

[AHHHZ975]

That's because I don't really know what are you training for, for example what task are we referring here.

When you say the fine-tuning process should guide us to a specific domain of text prompts that is more aligned with the fine-tuning objective, there won't be any problem if the fine-tuning objective is something the unet knows, for example generating images in a domain it was trained but if it's something the unet still doesn't know e.g.: medical pathologies (x-rays) or instruct pix to pix it won't work.

These are my assumptions though because I haven't trained something like this and haven't seen someone else do it.

Oh right, that totally makes sense to me now. But yes, my goal is to just search within the domain that the model was trained on before.

It's possible, there was a project called ELLA that did this, they released the code and weights for SD 1.5. Maybe that will work for what you need.

Interesting! Thanks for sharing it.

Yes, but in my experience textual inversions only work with concepts the model already knows, it doesn't make it learn new concepts and that's why they aren't as popular as LoRAs. Textual Inversion these days are mostly used to replace negative prompts or to guide the model to, for example, generate better hands (which in my experience never worked).

If you want to do something similar, in theory, it should also work.

Yes, it's true but I'd like to know, according to your experience, what general steps should be taken to generally get this done. , I am thinking that if methods like the prompt-tuning under the umbrella of the PEFT are working well in the area of LLMs, then I guess they should also work in the domain of text-to-image diffusion models (and generally computer vision) with some engineering tricks/implementations + solid developed theory. Am I right? I know that my question might be too general but I'd be happy to know your opinion on this and also what is the reason that methods like Textual Inversion do not work well.

[asomoza]

what is the reason that methods like Textual Inversion do not work well

Is not that Textual Inversion(TI) do not work well but more that the utility of them was superseded by LoRAs and better models, with SD 1.5 TI were used a lot, but when SDXL came, they lost the usefulness because SDXL mostly doesn't need negative prompts and you can't make the model do better generations with TI, you need to fine tune the model with millions of images.

Even more, with Dreambooth you can teach it to generate any kind of concept or subject, contrary to TI that can't teach it something new like for example how to generate a photo of yourself. Finally training LoRAs is really fast and cheap right now, even Flux which is a 12B model can be trained in a couple of minutes (online).

TI still has the utility with pivotal tuning, which is to prevent catastrophic forgetting of the previous concept, useful for people that use multiple loras of characters/persons.

if methods like the prompt-tuning under the umbrella of the PEFT are working well in the area of LLMs, then I guess they should also work in the domain of text-to-image diffusion models with some engineering tricks/implementations + solid developed theory ... what general steps should be taken to generally get this done

I still fail to understand what is your objective here, maybe @sayakpaul or @BenjaminBossan can answer better your question.

[BenjaminBossan]

See discussion here: huggingface/peft#2085