[1.1.431] Blend images with multiple IP-Adapters

[huchenlei]

See #2469, multiple IP-Adapter was broken because the original design of model caching mechanism assumes that model file name + model hash uniquely maps to a model object, but IP-Adapter model contains more information and cannot be uniquely idenfitied by file name + hash.

The easy fix that uncache IP-Adapter models has been landed. I will work on a more proper fix later, but finally we can use multiple IP-Adapter units to blend images.

Unit 0 setup

Unit 1 setup

Output

[sangoi-exe]

That's incredible! Thank you!

[dill-shower]

What we should use in prompt section for blend image?

[huchenlei]

You can use anything. In the example, I just left prompt empty.

[bordurodla]

how to update my ControlNet, which still on V1.1.411, and updated it failed by WebUI.

[huchenlei]

Can you file an issue on the WebUI error? Or you can enter controlnet directory, right click enter git bash and git pull manually.

[CarbonatedPastries]

Would this cause a significant spike in VRAM usage during the preprocessing stage? I keep my ControlNet up-to-date every day, and I was able to use IP-Adapters on January 11th, but today (January 17th) every time I try to run an IP-adapter I immediately get a spike of VRAM and a CUDA out-of-memory error which prevents the ControlNet from functioning. I only have 6GB of VRAM so this probably isn't noticeable on other systems. I do use Low VRAM mode.

[huchenlei]

Probay due to recent change that make clip inference run on CUDA: #2474

I will do some experiment to see if it is really leaking the vram.

[CarbonatedPastries]

Going over the discussion for that change, I see the change was moving CLIP processing from the CPU to the GPU(?). This means it simply wouldn't have worked for me through GPU due to limited VRAM, but was proven to work through CPU. In that case, would it be possible to make switching between GPU and CPU for CLIP a toggle for low-end users willing to wait (or maybe when Low VRAM is enabled)? Or would adding the toggle create other issues?

Edit: For the record, CLIP never worked on my GPU; It'd always do Out of Memory on my 1060 6GB. I can do other processes through my VRAM but CLIP is too much. Having the option to run the CLIP process through CPU for lower-mid range computers like mine would be nice, since by accident it was proven functional even if slow.

[huchenlei]

#2495