Outpainting with Diffusers

[asomoza]

Hi, I have made a 3 part series about outpainting with Diffusers, each one with a different method.

Outpainting I - Controlnet version

Outpainting using a controlnet trained for this. This method does change the original image but has the advantage that the ourpaining its more creative and seamless.

Outpainting II - Differential Diffusion

This method uses differential diffusion which allows us to use a soft inpainting mask that generates seamless outpainting, as the first method I can change the original image but not as much.

Outpainting III - Inpaint Model

The final method uses an inpainting model but it will preserve the original image so it's more suitable for professional work.

[tolgacangoz]

[asomoza]

fixed, thanks.

[crapthings]

control_image = Image.open(BytesIO(response.content))
new_controlnet_image = Image.new("RGBA", control_image.size, "WHITE")
new_controlnet_image.alpha_composite(control_image)

why this dones't work?

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[11], line 17
     14 b = midas(input_image)
     16 new_controlnet_image = Image.new('RGB', input_image.size, 'WHITE')
---> 17 new_controlnet_image.alpha_composite(ccc)
     18 # new_controlnet_image = Image.composite(input_image, new_controlnet_image, input_image)
     20 display(new_controlnet_image)

File [~/.local/lib/python3.10/site-packages/PIL/Image.py:1790](http://192.168.50.251:8888/lab/tree/workspace/ai-pipe/ai-pipe-replace-model-new/.local/lib/python3.10/site-packages/PIL/Image.py#line=1789), in Image.alpha_composite(self, im, dest, source)
   1787 else:
   1788     background = self.crop(box)
-> 1790 result = alpha_composite(background, overlay)
   1791 self.paste(result, box)

File [~/.local/lib/python3.10/site-packages/PIL/Image.py:3328](http://192.168.50.251:8888/lab/tree/workspace/ai-pipe/ai-pipe-replace-model-new/.local/lib/python3.10/site-packages/PIL/Image.py#line=3327), in alpha_composite(im1, im2)
   3326 im1.load()
   3327 im2.load()
-> 3328 return im1._new(core.alpha_composite(im1.im, im2.im))

ValueError: image has wrong mode

[asomoza]

When you're doing alpha composite you need both images to have an alpha channel. Probably the image you're loading as a control_image is a RGB image instead of a RGBA

[juzstu]

I find that the BrushNet also performs well in outpainting.

[asomoza]

Thanks, I have it under my radar, I'm waiting for the SDXL version but it seems that's going to take more time.

[juzstu]

@asomoza For your Outpainting III - Inpaint Model, I use lora to finetune the RealVisXL_V4.0, but the image quality was really bad, if not use the lora weights, the image quality turn to normal, can you tell me what case this and how can I use lora weights correctly to generate a normal quality image?

[asomoza]

That is a step by step guide, I provide a full script at the end but it's just so people can learn and follow it.

If you're just going to use it and change something, you should divide it into two parts, one for generating the temp background and the second part to actually generate the background with an inpainting model. Also you'll need to remove the fixed seeds from the code.

I'll need a little more information on what you're trying to do, are you loading the LoRA in the first step? the second? both? For what are you using the LoRA? the LoRA works ok if you just use it with a normal generation?

For example, if I just want to change the style of the generated background, I can just load the pixel art LoRA on the second step, and change the prompt for that one with the trigger for the LoRA.

source image	bg removed

temp background	final background	pixel art backgound

Hope the images explain the process a little better, but for me to be able to help you, I'll need more information on what you're trying to do.

[juzstu]

Thanks for your replay. I use the my own data to lora finetune the RealVisXL_V4.0 and use it in the first step, if I drop the zoe-depth model from the controlnets, the image will be normal. As you mentioned that the inpaint model likes to add details, but if I use zoe-depth or canny/depth mdel, the background only has pure color and no or just little details.

[asomoza]

In my last example I changed the source image, the prompts, the seeds and added a LoRA and it worked just fine, the controlnet works as intended even if you change those.

So the problem lies in one or a couple of what I mentioned, I'm guessing you're not sharing those because you can't, so I really can't help you anymore with the information you can give me.

My last suggestion is that you can try different settings for the depth controlnet, try lowering the controlnet_conditioning_scale or the control_guidance_end, maybe your LoRA needs more freedom to work.

[juzstu]

Thank you for your suggestion, I will give it a try.

[peki12345]

@asomoza I have carefully read these three plans and have been greatly inspired. Thank you very much.
However, I have to mention an important issue: when I only have text2img model, some lora or controlnet, and can't accept changes in no-mask areas, I can only use plan 2.
But after trying plan 2, I found that it could hardly complete the structure of the human body (object), let alone the face. Moreover, when the outpainting area is too large, the quality of generated content is low, and there may be problems connecting with no-mask areas.

For example, I used Plan 2 to double the repainting of this image, but the results I obtained were not very satisfactory.

But the result of Fooocus is always good

[madstuntman11]

Still have some transition artifacts, but it works for my task. Maybe the next step is to add ip-adapter with style block active.

[peki12345]

You're comparing big images with really small ones, if you want to achieve the same results as fooocus, you can scale down the image so SDXL can generate the background in one pass, that's what fooocus is doing. If you do that, you can use the third method with the inpaint model.

@peki12345 Tried it. Works good for my case (not photo realistic, but artistic). Just do not pad with simple white or black, fill with something reasonable. Model from the 3rd example will keep it white of black it this case. For example, use random edge tiles from original image.

This sounds reasonable, I will try it. But I don't know exactly how to do it yet. Can you provide a simple code?

[peki12345]

I was going to start working on this, but what always impresses me is how people don’t look closely at the fooocus results and think it’s really good when it isn’t.

Now that I look closer to your image, I can clearly see the seams and a lot of defects, changes in skin tones, patterns, it really makes the image not usable in any kind of serious work.

on the other side, the ones you generated with diffusers don't have the seams or the clear defects, it's a smooth transition but it doesn't do the person generation. Did you try with a prompt?

I will do the guide because I really think it's outdated even if it's a couple of months old, but there was a lot of improvements in this field. Still I think trying to achieve fooocus results is no longer a high standard, I think I can get better results with just inpainting in ComfyUI right now.

prompt

Yes, I must admit that fooocus has many defects. However, the seam appears abrupt because I directly pasted the original image without merge.

Regarding the issue of prompt, both fooocus and diffusers used a simple prompt that is independent of the image content. I believe a more detailed prompt will enhance the effect, but I want to use one prompt to infer all the images. I don't intend to create an interactive interface like fooocus, which is why I chose diffusers.

[asomoza]

but I want to use one prompt to infer all the images

I can tell you with that it will fail with all the complex images if you want to use a simple prompt for all the images. What you can do is to use a VLLM to generate the prompt without user intervention, but those 3 methods I posted will need a good prompt for complex images.

I'll try to achieve what you want in the new version but I can't be sure it will work, I have some ideas on how to do it.

Also, with this:

I directly pasted the original image without merge

It's been a while since I used fooocus, you mean inside fooocus or using a external software? I remember that fooocus didn't have that many options, you use the image and tell it to expand it or use a mask on a smaller image, all of my tests that I did with those options produced horrible results (similar to the images you posted) so I just assumed I was just bad at using it.

[peki12345]

It's been a while since I used fooocus, you mean inside fooocus or using a external software? I remember that fooocus didn't have that many options, you use the image and tell it to expand it or use a mask on a smaller image, all of my tests that I did with those options produced horrible results (similar to the images you posted) so I just assumed I was just bad at using it.

Using VLLM is indeed a good idea, but I need to check if my GPU memory is sufficient.
fooocus is only the main part, there are also some pre-processing and post-processing. In pre-processing, I used some tricks to make the obtained results more stable. If you use the original fooocus, there is a certain probability that you will receive horrible results, which is not your problem.