Fix images do not match / Coordinate 'right' is less than 'left'

[storyicon]

Description

I provide the webui as a cloud-based service to a large number of users, and I have been encountering this error frequently:

    Traceback (most recent call last):
      File "/data/app/aigc-worker/models/dev/webui-1.8.0/modules/call_queue.py", line 57, in f
        res = list(func(*args, **kwargs))
      File "/data/app/aigc-worker/models/dev/webui-1.8.0/modules/call_queue.py", line 36, in f
        res = func(*args, **kwargs)
      File "/data/app/aigc-worker/models/dev/webui-1.8.0/modules/img2img.py", line 235, in img2img
        processed = process_images(p)
      File "/data/app/aigc-worker/models/dev/webui-1.8.0/modules/processing.py", line 785, in process_images
        res = process_images_inner(p)
      File "/data/app/aigc-worker/models/dev/webui-1.8.0/modules/processing.py", line 855, in process_images_inner
        p.init(p.all_prompts, p.all_seeds, p.all_subseeds)
      File "/data/app/aigc-worker/models/dev/webui-1.8.0/modules/processing.py", line 1590, in init
        image_masked.paste(image.convert("RGBA").convert("RGBa"), mask=ImageOps.invert(self.mask_for_overlay.convert('L')))
      File "/root/miniconda3/envs/webui/lib/python3.10/site-packages/PIL/Image.py", line 1732, in paste
        self.im.paste(im, box, mask.im)
    ValueError: images do not match

(aka: Coordinate 'right' is less than 'left')

After parameter localization, I found two ways to reproduce the issue.

Reproduction Path 1

• Select img2img/Inpaint Upload
• Upload the image and mask separately, with the mask size being different from the image
• Select only masked in the inpaint area
• Click generate

Reproduction Path 2

• Use img2img / inpaint, upload the image and randomly paint a mask
• Change the resize to width or height, making it different from the uploaded image size
• Select only masked in the inpaint area
• In the script below, select SD upscale with the default 2x scaling
• Click generate

The specific code is located here code

class StableDiffusionProcessingImg2Img(StableDiffusionProcessing):
    ...
    def init(self, all_prompts, all_seeds, all_subseeds):
            ...
            if self.inpaint_full_res:
                self.mask_for_overlay = image_mask
                mask = image_mask.convert('L')
                crop_region = masking.get_crop_region(mask, self.inpaint_full_res_padding)
                crop_region = masking.expand_crop_region(crop_region, self.width, self.height, mask.width, mask.height)
                x1, y1, x2, y2 = crop_region

                mask = mask.crop(crop_region)
                image_mask = images.resize_image(2, mask, self.width, self.height)
                self.paste_to = (x1, y1, x2-x1, y2-y1)
            ...

Here, the self.mask_for_overlay is not resized to the size of self.width and self.height, leading to the following exception:

class StableDiffusionProcessingImg2Img(StableDiffusionProcessing):
    ...
    def init(self, all_prompts, all_seeds, all_subseeds):
            ...
            if image_mask is not None:
                image_masked = Image.new('RGBa', (image.width, image.height))
                image_masked.paste(image.convert("RGBA").convert("RGBa"), mask=ImageOps.invert(self.mask_for_overlay.convert('L')))
            ...

These users, when using the img2img/Inpaint Upload mode, have a slight difference in the size of init_img_inpaint and init_mask_inpaint, only a few percentage points. I guess they used some non-standard extensions or created the mask in a third-party tool, resulting in a slight difference in the size of init_mask_inpaint and init_img_inpaint.

Originally, I attributed this issue to user input errors, but recently, more and more users have reported this problem. Now I think it is a sufficient optimization item to improve the user experience of webui.

This change should be harmless, and the implementation of get_crop_region may indeed contain some typos.

This should be the same issue as described in #14331

Checklist:

• I have read contributing wiki page
• I have performed a self-review of my own code
• My code follows the style guidelines
• My code passes tests

[w-e-w]

some context

• when I rework get_crop_region in improve get_crop_region #14709

I specifically keep odd behavior exactly the same as before
even though it would cause an error if the box is not found in paint pipeline

in a way this function is a method of detecting if there's any non 0 (black) pixel in a image
the reason being that this function is also used by extensions
so it is possible that the extension is expecting the value to be invalid region (x1 > x2, y1> y2) as an indication that the image is all 0 and perform different actions based on it the this result

my priority at that point was compatibility, even though I don't know if any extension is relying on this behavior

if we want to play it safe it is better that in only masked inpaint mask region pipeline add a chack of invalid regions as opposed to changing the behavior of the function

but when performing only masked inpaint mask region, if you supply a mask is complete black, it essentially means that nothing needs to be changed, so functionally it doesn't matter if it errors or not
that's why I didn't add the check back then

[storyicon]

I had noticed this work (#14709) of yours when I submitted this PR but hadn't thought of it as a compatibility consideration, thanks for the tip, it's really interesting and I'll edit the submission here.

[storyicon]

some changes have been committed.

[w-e-w]

• alternative implementation Inpaint only mask switch to img2img if mask is blank #15579