Node graph implementation of diffusers

[asomoza]

Since I saw that there's interest in the Functional API posted by @tumurzakov, I was wondering if there's interest for a node conversion of diffusers. Since I use one under the hood, it wouldn't take much effort to make a separate module for it, simplify it and bring back some functionality that I butchered since I didn't need it.

Here's an example of how to use it:

node_graph = ImageArtisanNodeGraph()
node_graph.cpu_offload = True

sdxl_model = StableDiffusionXLModelNode(path="stabilityai/stable-diffusion-xl-base-1.0")
node_graph.add_node(sdxl_model)

vae_model = VaeModelNode(path="madebyollin/sdxl-vae-fp16-fix")
node_graph.add_node(vae_model)

positive_prompt = TextNode(text="photo of an astronaut in the jungle")
node_graph.add_node(positive_prompt)

prompts_encoder = PromptsEncoderNode()
prompts_encoder.connect("model", sdxl_model, "model")
prompts_encoder.connect("prompt_1", positive_prompt, "value")
node_graph.add_node(prompts_encoder)

latents = LatentsNode(width=1024. height=1024)
node_graph.add_node(latents)

scheduler = SchedulerNode(scheduler_index=1)
node_graph.add_node(scheduler)

image_generation = ImageGenerationNode()
image_generation.connect("scheduler", scheduler, "scheduler")
image_generation.connect("latents", latents, "latents")
image_generation.connect("embeds", prompts_encoder, "embeds")
image_generation.connect("model", sdxl_model, "model")
image_generation.connect("vae", vae_model , "vae")
node_graph.add_node(image_generation)

decoder = LatentsDecoderNode()
decoder.connect("latents", image_generation, "latents")
node_graph.add_node(decoder)

image_save = ImageSaveNode(path="images/")
image_save.connect("image", decoder, "image")
node_graph.add_node(image_save)

node_graph()

This approach enables the reutilization of resources or add new ones with ease, for example, you can add another model node and replace the connection to the image generator. The same for controlnets, T2I adapters and IP adapters, if you want to use them just add them to the graph and connect them, The system has nodes for all of them and they all can be used multiple times in the same graph.

I haven't put any effort in separating the API from the UI and make it public since even though I didn't use ComfyUI as a guide, I made a system that's very similar and probably any other system would be the same, so if you want to use a node system with UI you can just use ComfyUI, but maybe some people would have a use for a diffusers compatible node system.

I would appreciate any feedback on this.

[asomoza]

I forgot, this is not the same as an API or a pseudo language to use the pipelines, but my intention is not to compete or use another person ideas, so if this is somehow the same than "Functional API" proposed before or if It has a conflict with it, I don't have any problems in deleting this.

[tumurzakov]

Hello, nice work. I think there no difference what notation to use, functional or node or plain api. Same api could be used, notation is decorator for it.

[asomoza]

yeah you're right, after I wrote it I wanted to rephrase it but was too lazy, all of them are apis.

[tumurzakov]

How would you change dynamically parameters with node graph? For example i need separate prompt for each region by mask. That will need multiple unet iterations with different embeddings. Or even i need different controlnet for each iteration. How would your system deal with such difficult scenarios?

[asomoza]

I still haven't implemented the masks since I'm in the process of making the UI for them and inpainting, that takes way longer than the backend. But with a node system you don't have any restrictions, so it would be like this in pseudo code:

prompt1 = TextNode(text="photo of an astronaut in the jungle")
node_graph.add_node(prompt1)

mask1 = MaskNode(path="image.png")
node_graph.add_node(mask1)

prompts_encoder = PromptsEncoderNode()
prompts_encoder.connect("model", sdxl_model, "model")
prompts_encoder.connect("prompt_1", prompt1 , "value")
prompts_encoder.connect("mask", mask1 , "mask")
node_graph.add_node(prompts_encoder)

if you need multiple prompts and multiple masks, I would have to think about it, but that would probably need a new node to merge them. Each node can have multiple inputs in the same "connection" so instead of making it a single input they can be a list that you can manage inside each node separately.

I didn't post it before because with more nodes its harder to understand but for example to use multiple controlnets right now:

controlnet_model_one = ControlnetModelNode(path="controlnet-canny-sdxl-1.0-small")
node_graph.add_node(controlnet_model_one)

image_load_one = ImageLoadNode(path="canny.png")
node_graph.add_node(image_load_one)

controlnet_one = ControlnetNode(conditioning_scale=0.5, guidance_start=0.0, guidance_end=1.0)
controlnet_one.connect("controlnet_model", controlnet_model_one, "controlnet_model")
controlnet_one.connect("image", image_load_one, "image")
node_graph.add_node(controlnet_one)

controlnet_model_two = ControlnetModelNode(path="controlnet-depth-sdxl-1.0-small")
node_graph.add_node(controlnet_model_two)

image_load_two = ImageLoadNode(path="depth.png")
node_graph.add_node(image_load_two)

controlnet_two = ControlnetNode(conditioning_scale=1.0, guidance_start=0.0, guidance_end=1.0)
controlnet_two.connect("controlnet_model", controlnet_model_two, "controlnet_model")
controlnet_two.connect("image", image_load_two, "image")
node_graph.add_node(controlnet_two)

image_generation.connect("controlnet", controlnet_one, "controlnet")
image_generation.connect("controlnet", controlnet_two, "controlnet")

About the last question, I can think of multiple solutions, diffusers took a long time but the first thing I made was a callback in each step were you can change anything inside the node, so you can replace the controlnet or the embeddings for each step if you want, also there's another way and that would be to create multiple image generators with the controlnets, embeddings and the steps you want and then concatenate each one with the returned latents, I think that's what most people do in ComfyUI.

Even though I say this, my main focus is not creating this right now, this is just the backend of the UI I'm making so I haven't focused in complex scenarios like the ones you described and probably would never focus on video since IMO they're aren't really usable as professional (enterprise) solutions specially since I'm building a desktop app that relies on the single GPU of the user.

Its just that I saw some people with problems using multiple pipelines and requesting to add some features from one pipeline to other ones, this aren't issues in systems like yours or this one, so this probably can help for a while.

[sayakpaul]

cc: @patrickvonplaten @DN6

[patrickvonplaten]

I think a node-based implementation of diffusers could def make sense, but should be a seperate github repository no?

[asomoza]

yes, it was never my intention to do it in this repo, but it seems there's not that much interest for it or not a lot of people comes to this section, either of those.