add support for InstantX/FLUX.1-dev-Controlnet-Union

[geroldmeisinger]

Feature Idea

https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Union-alpha

Existing Solutions

No response

Other

No response

[geroldmeisinger]

https://www.reddit.com/r/StableDiffusion/comments/1euaeum

[hartmark]

Awesome, it seems each day in AI is like a year in other technologies.

Looking forward to this.

I tried using an image with low denoise with flux and sometimes the results were good but it was after tweaking the prompt and denoise value for each photo I wanted to generate.

[EeroHeikkinen]

Now that #4444 has been merged you can load InstantX's union controlnet using this custom loader node: https://github.com/EeroHeikkinen/ComfyUI-eesahesNodes
Workflow
You can find more detailed instructions here:
https://www.reddit.com/r/StableDiffusion/comments/1euz2a9/union_flux_controlnet_running_on_comfyui_workflow/

[geroldmeisinger]

it works great, thank you so much!
memory requirements are quite high though. I wasn't able to create a 1024x1024 image with all flux and t5 in fp8 and even turning off all desktop managers and browser to free VRAM. I guess we need fp8 controlnets too.

[hartmark]

it works great, thank you so much! memory requirements are quite high though. I wasn't able to create a 1024x1024 image with all flux and t5 in fp8 and even turning off all desktop managers and browser to free VRAM. I guess we need fp8 controlnets too.

How much vram did you have?

[geroldmeisinger]

16 VRAM (about 200M occupied by DM), 64 SRAM, 4 swap

[geroldmeisinger]

10a79e9
ea3f39b
b33cd61

[rootliuat]

The downloaded instantX flux union controller loader model should be placed in that directory