[Feature] Keep Original Mask Contents

[mpmo10]

Describe the bug
When I try to inpaint an image and I make the square more than 512px, the image inside the square becomes lower res. I like making the "D" (Dream) square bigger because the bigger it is, the better the inpainting will be since the AI will have a clearer understanding of the whole image.

To Reproduce
Steps to reproduce the behavior:

Upload an image, inpaint part of it with the Mask Brush (M) tool, make the generation Dream (D) square 1024px (or any res higher than 512px, but 1024px is the most obvious) instead of the default 512px square by moving the mouse scroll wheel upwards, generate an image and compare before and after of what's inside the box.

Expected behavior
For the image that's not been masked and that's inside the "D" Dream square box not to become lower res and stay the same after generation

(First image is before generation, second image is after). See how the stars and city lights are lower res. This doesn't happen when the square is 512px. Even if I increase the square, the unmasked image inside the square shouldn't be affected but as you can see it is being affected

[zero01101]

i suspect you're running into the decoupled reticle/SD resolution, which is actually by design but has a PR and related issue which optionally syncs the resolution to the cursor and vice versa.

with the current main-branch version (v0.0.9.6), if you expand the stable diffusion settings and set the resolution value to match your reticle size, does the output match your expectations?

[edit]
reread your issue and realized i might have missed the more important point, but just tried and was unable to confirm locally - all unmasked regions stay exactly the unmodified resolution that they're meant to...

[mpmo10]

Try inpainting a 1920x1080 image. Then choosing 1024px square, inpaint, then download the canvas. Now open the original and the new canvas images in your pc’s photo gallery and quickly switch between the two and focus on the image where the square was. Do you not see a lower res in the new canvas compared to the original? Download the two images I uploaded and quickly switch between them using your pc’s photo viewer, you’ll see a clear degradation in quality for the new canvas

[seijihariki]

Have you changed generation resolution as well as cursor size when inpainting?

[zero01101]

[instructions] Do you not see a lower res in the new canvas compared to the original?

honestly, i sure don't in my output :/ no argument that there's a significant difference in your screenshots at all however

first image was an imported 1920x1080 screenshot just simply as-is using the save canvas button, second was, uh, after turning the opponent into a ""football"" with the color brush, third is a pathetic attempt to get a hot dog out of the oppopnent using the mask brush lmao

[mpmo10]

Have you changed generation resolution as well as cursor size when inpainting?

You mean this right? I get horrible inpainting results if I increase this value. I think because the inpainting model has been trained on 512x512 images so it's best not to change this value (try keeping a fixed seed and compare the generations between 512 vs 768 or 1024 res and you'll see that 512 is the best by far)

[mpmo10]

[instructions] Do you not see a lower res in the new canvas compared to the original?

honestly, i sure don't in my output :/ no argument that there's a significant difference in your screenshots at all however

first image was an imported 1920x1080 screenshot just simply as-is using the save canvas button, second was, uh, after turning the opponent into a ""football"" with the color brush, third is a pathetic attempt to get a hot dog out of the oppopnent using the mask brush lmao

Yeah there's no degredation in quality for the pics you uploaded. Hmm. Are you sure you increased the Dream (D) canvas size from 512px to higher (while keeping the resolution, which is under 'Stable Diffusion Settings' at 512 resolution) ?

[mpmo10]

Are you sure you're increasing the canvas size? Watch this vid, this is what I'm doing exactly: https://streamable.com/zv7wpi

I attached before generation and after generation pics below. You can see a clear degredation in quality of the stuff that was inside the box (download the two pics and quickly switch between the two to see the difference)

Before:

After:

[seijihariki]

Have you changed generation resolution as well as cursor size when inpainting?

You mean this right? I get horrible inpainting results if I increase this value. I think because the inpainting model has been trained on 512x512 images so it's best not to change this value (try keeping a fixed seed and compare the generations between 512 vs 768 or 1024 res and you'll see that 512 is the best by far)

Yeah, so that's the thing... We cannot really keep the generation resolution the same and increase the cursor without losing quality in the process, as we are effectively generating the image in 512x512 and scaling it up to the final resolution using just normal image scaling.

If you are refering to the unmasked regions (that should not be modified) we can see and try a fix, manually keeping these areas.

But for the actual region expected to be modified/inpainted, keeping same resolution as input is impossible, unless we we run the upscaling function in the mix. But at that point upscaling should actually be its own tool.

[mpmo10]

If you are refering to the unmasked regions (that should not be modified)

"If you are refering to the unmasked regions (that should not be modified)" yup that's exactly what i'm referring to. I understand the masked region will be a little lower res and that's fine, but if you can add an option to keep the unmasked regions completely the same that would be perfect!

[seijihariki]

Maybe you should give full resolution inpainting a try, but as in this case it has no context outside of the area being inpainted, you need to be more specific with your prompts.

[seijihariki]

If you are refering to the unmasked regions (that should not be modified)

"If you are refering to the unmasked regions (that should not be modified)" yup that's exactly what i'm referring to. I understand the masked region will be a little lower res and that's fine, but if you can add an option to keep the unmasked regions completely the same that would be perfect!

That makes sense... Will check what it would take to implement this. Probably will only really apply to fully opaque masking to avoid dealing with averaging the images. Would that be okay to you?

[mpmo10]

If you are refering to the unmasked regions (that should not be modified)

"If you are refering to the unmasked regions (that should not be modified)" yup that's exactly what i'm referring to. I understand the masked region will be a little lower res and that's fine, but if you can add an option to keep the unmasked regions completely the same that would be perfect!

That makes sense... Will check what it would take to implement this. Probably will only really apply to fully opaque masking to avoid dealing with averaging the images. Would that be okay to you?

Yup that would be good, thanks a lot

[seijihariki]

This is not a bug anymore, but a feature request. We have to implement a feature to reapply masked surfaces to the image received from the remote before displaying them to the user (probably controlled by a checkbox or something). - Keep-masked maybe?

[zero01101]

yeah, i definitely missed the context of 512px render resolution with the larger reticle, my fault on that one, and agreed it shouldn't replace anything that isn't masked, but yeah, as mentioned, stable diffusion itself is just working off of a 512x512 area at that point and we're simply replacing the previous high-res content with the new low-res response...

as a crappy workaround for now, you could put the wrong-resolution masked inpaint/img2img on a new layer and erase away the undesirable bits?

[seijihariki]

Hi, @mpmo10, can you check if the implementation is okay for this feature? It's on the testing branch (git checkout testing).

An option was added, and is enabled by default, where all fully masked pixels on the original image are kept.

[mpmo10]

Hi, @mpmo10, can you check if the implementation is okay for this feature? It's on the testing branch (git checkout testing).

An option was added, and is enabled by default, where all fully masked pixels on the original image are kept.

Tried it just now, yes perfect! thanks a lot. But I do notice that when I have "Keep Masked" off, it does a better job at blending in the edges of the masked with unmaked. But when the "Keep Masked" is turned on, you can see the lines of where the mask brush has been more clearly. Is there a way to make it blend in more when "Keep Masked" is turned on? I tried increasing the "Mask blur" to 20 but even at that high value the edges of the masked and unmasked parts don't blend in that well as when "Keep Masked" is off. And going higher than 30 Mask blur makes the inpainting generations worse for some reason

[seijihariki]

Well, that's kind of to be expected, as we have a very crisp original image right beside an area of the image that was generated at low res.

We can try adding a keep image blur slider if you want, but it will be a simple gaussian blur, so I do not guarantee that it will be anywhere near good.

[mpmo10]

Well, that's kind of to be expected, as we have a very crisp original image right beside an area of the image that was generated at low res.

We can try adding a keep image blur slider if you want, but it will be a simple gaussian blur, so I do not guarantee that it will be anywhere near good.

Yeah a gaussian blur slider would be great 🙏🏼 (even better if changing the slider would change the image in real time)

[seijihariki]

If you have any other ideas for implementation, please tell us so we can try!

For now, will implement a slider for blur, but that's the only thing that we can do for now.

[seijihariki]

Yeah a gaussian blur slider would be great 🙏🏼 (even better if changing the slider would change the image in real time)

In real time... would be kind of difficult, considering now we have multiple concurrent generations/image picking. This pull will include a non-realtime implementation, but will see what I can do in the future.

[seijihariki]

Check now the testing branch.

[mpmo10]

Check now the testing branch.

Is it supposed to be here? I've selected the Dream (D) button, not the img2img (i) button

[seijihariki]

Can you open the console, disable caching and reload the page? It should have a slider under keepMasked.

[seijihariki]

I imagine you ran git pull already.

[mpmo10]

I imagine you ran git pull already.

Aha ok I see it. Yess it does work. Thanks for adding this feature!!