[API-UPDATE] Need to update to accommodate recent API changes in AUTOMATIC1111

[PhoenixCreation]

What Happened

• Add a way for API txt2img and img2img requests to pass args to always on scripts AUTOMATIC1111/stable-diffusion-webui#8187 got merged recently.
• This allows user to use Controlnet and other scripts from the /sdapi/v1/txt2img and sdapi/v1/img2img with parameter alwayson_scripts.
• This will allow us to now deprecate the /controlnet/txt2img and /controlnet/img2img routes as mentioned in wiki.

Issue 1

• I haven't understood entire flow of the AUTOMATIC1111's API but I will try my best to explain this.
• Currently ScriptRunner class has process method which is responsible to pass on arguments to the other scripts like Controlnet.
• ScriptRunner.process will call Script.process method with expected method signature of process(self, p, *args): which will allow all scripts to be run with API.
• For Controlnet script's Script.process method has different signature process(self, p, is_img2img=False, is_ui=False, *args) which is causing trouble if we want to just directly use the Controlnet from the /sdapi/v1/txt2img API because first two parameters will go towards the is_img2img and is_ui.
• So current schema for the Controlnet parameters will become as following for solution of this issue:

"ControlNet": [
  false,  # is_img2img
  false,  # is_ui
  true,   # enabled,
  "canny",    # module,
  "control_canny-fp16 [e3fe7712]",    # model,
  1,    # weight,
  {"image":"BASE64_IMAGE_STRING"},    # image,
  false,    # scribble_mode,
  "Scale to Fit (Inner Fit)",   # resize_mode,
  false,    # rgbbgr_mode,
  false,    # lowvram,
  512,    # pres,
  100,    # pthr_a,
  200,    # pthr_b,
  0,    # guidance_start,
  1,    # guidance_end,
  false   # guess_mode
]

Ref:

sd-webui-controlnet/scripts/controlnet.py

Line 610 in 5bf7e54

enabled, module, model, weight, image, scribble_mode, \

This is current workaround and I am mentioning this because Second issue is dependent on this.

Curl Command for the Request :https://pastebin.com/432EG3PT

Issue 2

• If you go ahead and send the curl request mentioned above then you will encounter KeyError for 'image' for following line of code:

  sd-webui-controlnet/scripts/controlnet.py

  Line 751 in 5bf7e54

  input_image = HWC3(image['image'])

• Basically image['image'] is currently string not numpy array of image.
• This is also mentioned in the description of the Add a way for API txt2img and img2img requests to pass args to always on scripts AUTOMATIC1111/stable-diffusion-webui#8187

• For exemple, controlnet might have to detect that the images are a base64 string instead of an Image but it shouldn't require much work.

• I will be raising PR for this detection check. But till then it won't work. Edit: Raised PR(Add str check for image for API compatibility related to #567 #568)

I am creating this issue so we can keep track of the API related changes. Also let me know if Issue 1 needs to be changing but that is something you should handle as you have more knowledge of the API flow.

Tasks

• Raise PR for image str detection(Add str check for image for API compatibility related to #567 #568)
• Mark APIs /controlnet/txt2img and /controlnet/img2img as deprecated.
• Update wiki accordingly. [I guess this needs to be done by owner]

[Vespinian]

For posterity, at time of writing and to my understanding, the way args for scripts work in auto1111 is that there is a giant list of args that cover all scripts. Each script gets handed a args_from and args_to variable to know where their args are in that list. One of the things I did in my PR is to initialize that list with None everywhere and then fill it in with whatever is passed in the API request instead of not generating one at all or a very limited one when using selectable scripts. After initializing the arg list with None in every position to the correct size to cover all scripts, the API request finds the script you specified in the alwayson_scripts dict, checks if it has args in the request and places those args at their intended position in the big arg list.

If I recall correctly, in the case of controlnet, the number of args will depend on the max controlnet modules you have. While only filling one module, if you have more then one, might not be a problem, you can essentially fill everything. So the controlnet API might benefit from a get config or get max_model api route, if it hasn't already got one, which would allow the client to know how many controlnet modules he can pass and the schema in the docs might want to account for this.

[PhoenixCreation]

If I recall correctly, in the case of controlnet, the number of args will depend on the max controlnet modules you have. While only filling one module, if you have more then one, might not be a problem, you can essentially fill everything. So the controlnet API might benefit from a get config or get max_model api route, if it hasn't already got one, which would allow the client to know how many controlnet modules he can pass and the schema in the docs might want to account for this.

@Vespinian, I understood that concept but the issue is that two args(is_img2img and is_ui) are already there in process method of controlnet script which throws of the count of arguments to be passed on.

Currently each controlnet model takes 15 args(ref:

sd-webui-controlnet/scripts/controlnet.py

Line 72 in 5bf7e54

PARAM_COUNT = 15

) and so basically one should only need to provide 15 arguments to run single controlnet and 30 arguments to run two controlnet and so on. But due to the predefined args(is_img2img and is_ui) you need to pass two extra args in first and second element of arrays for them manually. so total args count for 1 controlnet will become 17, for two it would be 32 and so on. Which I think is okay, but can be refactored to be removed.

But this is great thing as it will allow all AUTOMATIC1111 scripts to be streamlined into single flow format.

[killamonis]

The curl command linked above is unable to be viewed ("Error, this is a private paste or is pending moderation. If this paste belongs to you, please login to Pastebin to view it.").

I've tried to use the alwayson_scripts with the sdapi/v1/txt2img route and I'm still not getting images using responses, not sure if I'm formatting things incorrectly.

[Vespinian]

Exemple alwayson_scripts, the 2 inline args are is_img2img and is_ui, the rest are the 15 args for the controlnet, if you want to use multiple nets you just append another 15 args in the list for each module.

    "alwayson_scripts": {
      "ControlNet": {
        "args": [false, false,
            true,
            "none",
            "difference_controlSd15Openpose [1723948e]",
            1.0,
            {"image": "base64img..."},
            false,
            "Scale to Fit (Inner Fit)",
             false,
             false,
             64,
             64,
             64,
            0.0,
            1.0,
            false
            ]
        }
    }

Workaround for now since the goal of this update is to remove the need to pass is_img2img and is_ui

[killamonis]

@Vespinian
tried running the API call with that format and got this error

422
api call made
{'detail': "Script 'ControlNet' not found"}

My call body

"prompt": prompt,
            "negative_prompt": "",
            "seed": -1,
            "subseed": -1,
            "subseed_strength": 0,
            "batch_size": 1,
            "n_iter": 1,
            "steps": 20,
            "cfg_scale": 7,
            "width": 512,
            "height": 512,
            "restore_faces": True,
            "eta": 0,
            "sampler_index": "Euler a",
            "alwayson_scripts": {
                "ControlNet": [
                    False,  # is_img2img
                    False,  # is_ui
                    True,   # enabled,
                    "canny",    # module,
                    "control_sd15_canny [fef5e48e]",    # model,
                    0.3,    # weight,
                    {"image":[b64img]},    # image,
                    False,    # scribble_mode,
                    "Scale to Fit (Inner Fit)",   # resize_mode,
                    False,    # rgbbgr_mode,
                    False,    # lowvram,
                    512,    # pres,
                    100,    # pthr_a,
                    200,    # pthr_b,
                    0,    # guidance_start,
                    1,    # guidance_end,
                    False   # guess_mode
                ]
            }

[killamonis]

When I use controlnet through the UI it works

[ljleb]

related: #527 and #528

Didn't think auto would merge the alwayson scripts PR so quickly. in #527 I tried changing how the args of gradio are passed so that we can accept dicts instead of an unreasonably long list of arguments to keep the api as close to what it is right now. I'll try spending a bit of time on this to see if it can be a good fix to the situation.

[PhoenixCreation]

Hi @killamonis, Can you confirm following things?

• Have you updated the controlnet extension and AUTOMATICA1111 to latest version? If yes, Can you please provide last commit id for both of them?
• If you have updated both of them, run the curl command from https://pastebin.com/432EG3PT, and let us know if it still gives the same error.

[ljleb]

@Vespinian If I recall correctly, in the case of controlnet, the number of args will depend on the max controlnet modules you have.

That is only true for the gradio interface. In the web API (and also external code API), you can pass an arbitrary number of control net units and the webui will resize the script args of the controlnet script to match the passed args.

Edit: I looked at the code again, and it seems that the webui /sdapi/v1/*2img routes do not resize the args actually.

[Vespinian]

Alright. Also I believe that in the alwayson scripts PR, the way the arg list is initialized is currently overwriting the modifications made by the scriptrunner hijack, which broke the API hijack, see #571. I made a bit of a dirty fix in a branch of my fork by taking the control units from the request and creating a ControlNet dict in the request's alwayson_scripts but that may have broken other things.

[ljleb]

Yes, thanks for your quick fix. I opened #527 also for this purpose. Hopefully I can get some confidence on the code and will open for review when we get there.

[killamonis]

Hi @killamonis, Can you confirm following things?

• Have you updated the controlnet extension and AUTOMATICA1111 to latest version? If yes, Can you please provide last commit id for both of them?
• If you have updated both of them, run the curl command from https://pastebin.com/432EG3PT, and let us know if it still gives the same error.

Hey @PhoenixCreation
Updating the controlnet extension allowed me to actually send the controlnet request, but when I send it through the API, I'm getting this error, and the generated image doesn't use the controlnet pose.

Automatic commit: AUTOMATIC1111/stable-diffusion-webui@dfeee78

Controlnet extension commit: 274dd5d

[killamonis]

However, using the curl command you sent and the UI, the controlnet pose does work.

[freecoderai01]

I'm unable to run the extension in img2img mode. Not sure if related to current issue.
(tested on latest commit of automatic, AUTOMATIC1111/stable-diffusion-webui@a9fed7c)
The output with error

Error running process: /content/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py
Traceback (most recent call last):
  File "/content/stable-diffusion-webui/modules/scripts.py", line 409, in process
    script.process(p, *script_args)
  File "/content/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 737, in process
    resize_mode = resize_mode_from_value(resize_mode)
  File "/content/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 184, in resize_mode_from_value
    return ResizeMode(value)
  File "/usr/lib/python3.9/enum.py", line 384, in __call__
    return cls.__new__(cls, value)
  File "/usr/lib/python3.9/enum.py", line 702, in __new__
    raise ve_exc
ValueError: 'OUT' is not a valid ResizeMode

[ljleb]

Hi @freecoderai01, can you elaborate how you interact with the extension? The value 'OUT' is not a valid resize mode, I assume this was passed directly as "resize_mode": "OUT" in the API? If that is the case, consider using int value 2 or the string "Envelope (Outer Fit)" as documented in the wiki.

Otherwise, please clarify what situation caused the error.

[freecoderai01]

Hi @freecoderai01, can you elaborate how you interact with the extension? The value 'OUT' is not a valid resize mode, I assume this was passed directly as "resize_mode": "OUT" in the API? If that is the case, consider using int value 2 or the string "Envelope (Outer Fit)" as documented in the wiki.

Otherwise, please clarify what situation caused the error.

Hi @ljleb, I was not trying to use the extension's API, so the value should be in the default state, isn't it?
For context, I was interacting via automatic1111 webui (running inside google colab, without --api arg, full list of args:
--disable-safe-unpickle --enable-insecure-extension-access --share --gradio-img2img-tool editor --theme dark --ckpt-dir /content/stable-diffusion-webui/models/Stable-diffusion --gradio-queue --disable-console-progressbars --no-hashing --opt-channelslast --xformers),
img2img tab with controlnet subtab. Gradio: 3.16.2. While using web browser, switching between Just Resize/Scale to Fit (Inner Fit)/Envelope (Outer Fit) did not affect the problem.

[ljleb]

Thanks for the context. I think this should go in its own issue, it does not seem related to API migration.

[mengchengwanli]

这样做可能会遇到的一个潜在问题是，如果用户将 controlnet 脚本替换为脚本运行器中的副本。我不认为这是一个阻塞问题，因为它可能不会发生，但它可能会导致令人惊讶的行为。

在未更改的 SD 安装上进行常规 api 调用时，会发生这种情况吗？我不完全了解这个问题以及它将如何影响脚本的行为，但是一旦我完成了重新映射对象和迁移调用，我可以帮助测试内容。


合作者
作者
ljleb 评论 2 days ago •
编辑

This parameter error, do not understand why;
Using the original parameter passing mode can be achieved;

[benrugg]

For anyone coming here in the future, here is the new format for how to use the Automatic1111 alwayson_scripts parameter in /sdapi/v1/img2img:

https://github.com/Mikubill/sd-webui-controlnet/wiki/API#migrating-from-controlnet2img-to-sdapiv12img