Super-resolution conditional training

[CallShaul]

Hi,

How can I use the provided repository for image-conditional training and evaluation (super-resolution)?
And which approach is recommended for such a task? consistency-distillation? or consistency-training?

(Assuming the base model for the consistency-distillation originates from: "Image Super-Resolution via Iterative Refinement" )

Thanks!

[nekoshadow1]

Hi Shaul,

You can use consistency-distillation if you have a well-trained teacher model. However, keep in mind that the performance of consistency distillation depends on the performance of your teacher model. On the other hand, you can go ahead with consistency training. However, consistency training converges slower than consistency distillation.

Notice that your choice of teacher model is super resolution, a non-DDM model. However, in this repo, the author uses a DDM as the default teacher model in consistency distillation. And it is class-conditional, not image-conditional. Therefore, to make it work in your case, you need to modify the inputs of the consistency training and/or consistency distillation to accept image as condition instead of class. Then replace the DDM teacher with your choice of teacher model, it should work. It took me a while to make it work in my image-to-image task, but it's feasible. Hope this helps!

[aarontan-git]

@nekoshadow1 could you share your image-to-image code?

[CallShaul]

Hi @nekoshadow1
I have adjusted K-diffusion to support img2img ()
So now I want to move on to distill this img2img model to fit fast inference using this Consistency model repo.

Note that img2img / super-resolution is practically the same img2img architecture, only trained differently,
so why you've said that it is "non-DDM model" ?

Any tips for that next stage, or even sharing your code would be highly appreciated. Thanks !