An alternative inpainting model that supports inpainting, removal and outpainting. It can be can be used with controlnet

[zengyh1900]

Hi all,

I want to share our recent model for image inpainting, PowerPaint.
It can be the most powerful inpainting model that enables text-guided object inpainting, text-free object removal, and image outpainting, as an open-source alternative of Adobe Firefly.

• youtube: https://youtube.com/watch?v=7QsiY1JQUfg
• online demo: https://openxlab.org.cn/apps/detail/rangoliu/PowerPaint
• arxiv: https://arxiv.org/abs/2312.03594
• code: https://github.com/open-mmlab/mmagic/tree/main/projects/powerpaint

We are looking forward to any feedback!

[zengyh1900]

Notably, PowerPaint can be used with ControlNet (e.g., canny edge, pose, etc.)!

[zengyh1900]

Compared with Adobe Firefly,

[zengyh1900]

Some more results here. Notably, all these are achieved by using only a single model without tailored workflow!

[huchenlei]

Reference discussion: Acly/krita-ai-diffusion#221

[lllyasviel]

is the model XL or 1.5?

[huchenlei]

Our method does have two inherent limitations. Firstly,
the synthesis quality can be constrained by the capabilities
of the underlying text-to-image diffusion model. We plan to
alleviate this limitation by extending our method to a more
powerful text-to-image diffusion model, such as SDXL [22]

Looks like SD1.5.

[sdbds]

I think they use diffusers key

[sdbds]

i transform it into sd model,now we can use it in fooocus、comfyui and webui
https://huggingface.co/bdsqlsz/PowerPaint-V1-stable-diffusion-inpainting

[Woisek]

And how is it now used? 🤔

[sdbds]

use as stable diffusion inpainting model,but their shaped-guide use fitting degree not implement in webui now

[Woisek]

Does this mean, it's still not usable ... ? And how about the Photoshop Plugins? Do they require something special to use this as inpainting model?

[sdbds]

They actually had one way to use shaped guide and extra arg called fitting degree to use.
But you can use normal way such as text inpaint、remove something and outpainting.

[Woisek]

That sounds cool then. Will have a look. Thanks! 👍

[dilectiogames]

can someone answer like I'm 5years old if this can be used with my local automatic installation and what I need to do to make this work.

do I need to download a model for controlnet or what do I need to do?

[zhuang2002]

PowerPaint Update

We've released an updated model with enhanced stability, now available for interactive testing on our online demo.

Paper: https://arxiv.org/abs/2312.03594
HomePage: https://powerpaint.github.io/
Online Demo: https://openxlab.org.cn/apps/detail/rangoliu/PowerPaint
Code: https://github.com/open-mmlab/mmagic/tree/main/projects/powerpaint

[Woisek]

@sdbds Would you again be so kind and convert this for us, please? 🙂
@zhuang2002 It would be a good idea, to (also) release a converted safetensor file for the community for those who are not able to convert this themselves. 👍 Then we all could appreciate this nice work even more.

[sdbds]

updated now
https://huggingface.co/bdsqlsz/PowerPaint-V1-stable-diffusion-inpainting/

[Woisek]

Many thanks! 👍

[Fishkin]

Can't get this to work. I mean, it "works" except it generates completely unrelated stuff. I just use it as model and inpaint normally, do I have to do something extra? ControlNet inpainting model on top of it doesn't help much either.

[soldivelot]

i made it work my merging with another base model

[IndigoDosSantos]

Shape-guided object inpainting is not yet supported in webui, or is it?

[huchenlei]

You can use ControlNet during inpainting. I am not sure what do you mean "shape"? (Depth? Canny?)

[IndigoDosSantos]

PowerPaint was trained on three task prompts:

• P_obj for text-guided object inpainting
• P_ctxt for context-aware image inpainting
• P_shape for shape-guided inpainting (If I've understood it correctly...)

In the gradio demo of PowerPaint there is a slider that controls how closely the inpainted object aligns with the shape of the mask and supposedly works without any additional ControlNet conditions.

[alexgilseg]

Is this usable with A1111?

[IndigoDosSantos]

Yes, it is.

[zhuang2002]

We have recently developed a new version of PowerPaint, drawing inspiration from Brushnet. The Online Demo has been updated accordingly. In the coming days, we intend to make the model weights available as open source.

The PowerPaint model possesses the ability to carry out diverse inpainting tasks, such as object insertion, object removal, shape-guided object insertion, and outpainting. Notably, we have retained the cross-attention layer that BrushNet had removed, which is essential for task prompt input.

	Object insertion	Object Removal	Shape-guided Object Insertion	Outpainting
Original Image
Output