Add a way for API txt2img and img2img requests to pass args to always on scripts #8187

Vespinian · 2023-02-28T05:19:29Z

Describe what this pull request is trying to achieve.

UPDATE:
Changed to add 1 param, a dictionary, to the api for txt2img and img2img (alwayson_scripts) in the format proposed in the comment below.

~~These changes add 2 params to the api for txt2img and img2img (alwayson_script_name, alwayson_script_args).~~

alwayson_script_name is a list of string containing the name of the always on scripts you want to pass args to
exemple: ["Additional networks for generating", "ControlNet"]

alwayson_script_args is a list of lists containing args for script mentioned in alwayson_script_name
exemple: "alwayson_script_args": [[true, false, "LoRA", "None", 1, 1, "LoRA", "None", 1, 1, "LoRA", "None", 1, 1, "LoRA", "None", 1, 1, "LoRA", "None", 1, 1, "Refresh models"] , [!Controlnet args here!]]

The goal is to pass args to always on scripts like controlnet or additional-networks without them having to duplicate or hijack parts of the API just to have access to them. Some adjustment on the part of the extensions might be needed if they want to support it. For exemple, controlnet might have to detect that the images are a base64 string instead of an Image but it shouldn't require much work.
To test this, I had to do a small change to additional_network because it would throw an error. I believe it was checking the 0 index of a possibly empty tuple to see if it existed.
These changes should not affect how the api works currently, selectable scripts should be working exactly like before.

Environment this was tested in

List the environment you have developed / tested this on. As per the contributing page, changes should be able to work on Windows out of the box.

OS: Windows
Browser: Chrome
Graphics card: NVIDIA RTX 2080 8GB

Exemple txt2img request with additional network as an alwayson_script:

{
	"enable_hr": false,
	"denoising_strength": 0,
	"firstphase_width": 0,
	"firstphase_height": 0,
	"hr_scale": 1,
	"hr_upscaler": false,
	"hr_second_pass_steps": 0,
	"hr_resize_x": 0,
	"hr_resize_y": 0,
	"prompt": "a prompt",
	"styles": [],
	"seed": -1,
	"subseed": -1,
	"subseed_strength": 0,
	"seed_resize_from_h": -1,
	"seed_resize_from_w": -1,
	"sampler_name": "DPM++ 2S a",
	"batch_size": 1,
	"n_iter": 1,
	"steps": 10,
	"cfg_scale": 7,
	"width": 512,
	"height": 512,
	"restore_faces": false,
	"tiling": false,
	"negative_prompt": "",
	"s_churn": 0,
	"s_tmax": 0,
	"s_tmin": 0,
	"s_noise": 1,
	"override_settings": {},
	"override_settings_restore_afterwards": true,
	"script_name": "",
	"script_args": [],
	"alwayson_scripts": {
		"Additional networks for generating": {
			"args": [true, false, "LoRA", "<!!!!!A lora model here!!!!>", 0.3, 0.3, "LoRA", "None", 1, 1, "LoRA", "None", 1, 1, "LoRA", "None", 1, 1, "LoRA", "None", 1, 1, "Refresh models"]
		}
	}
}

closes #8253

Added 2 additional possible entries in the api request: alwayson_script_name, a string list, and, alwayson_script_args, a list of list containing the args of each script. This allows us to send args to always on script and keep backwards compatibility with old script_name and script_arg api params

AUTOMATIC1111 · 2023-03-11T10:58:19Z

I'd prefer if it was a dictionary like this:

	"alwayson_scripts": {
		"Additional networks for generating": {
			"args": [true, false, "LoRA", "<!!!!!A lora model here!!!!>", 0.3, 0.3, "LoRA", "None", 1, 1, "LoRA", "None", 1, 1, "LoRA", "None", 1, 1, "LoRA", "None", 1, 1, "Refresh models"]
		},
		"ControlNet": {
			"args": [!Controlnet args here!]
		}
	}

… alwayson_scripts param dict

…usion-webui

Vespinian · 2023-03-11T19:45:10Z

Changed to the proposed format and updated the PR description.

mengchengwanli · 2023-03-16T14:21:00Z

我更喜欢这样的字典：

	"alwayson_scripts": {
		"Additional networks for generating": {
			"args": [true, false, "LoRA", "<!!!!!A lora model here!!!!>", 0.3, 0.3, "LoRA", "None", 1, 1, "LoRA", "None", 1, 1, "LoRA", "None", 1, 1, "LoRA", "None", 1, 1, "Refresh models"]
		},
		"ControlNet": {
			"args": [!Controlnet args here!]
		}
	}

报错！ controlnet 参数根本不知道咋传，全靠猜，还不对！

Vespinian · 2023-03-16T14:28:11Z

@mengchengwanli
Look at the Controlnet Wiki

mengchengwanli · 2023-03-16T15:57:03Z

@Vespinian
Here I see it, now I'm using it like this;
However, the reference here is the deprecated mode and the new mode does not take effect;

Vespinian · 2023-03-16T16:08:46Z

The examples in the Migrating from /controlnet/*2img to /sdapi/v1/*2img works for me using the /sdapi/v1/*2img routes. You do need to have both the webui and the controlnet extension up to date though.

mengchengwanli · 2023-03-17T01:24:50Z

@Vespinian

"alwayson_scripts": {
      "ControlNet": {
        "args": [false, false,
            true,
            "none",
            "difference_controlSd15Openpose [1723948e]",
            1.0,
            {"image": "base64img..."},
            false,
            "Scale to Fit (Inner Fit)",
             false,
             false,
             64,
             64,
             64,
            0.0,
            1.0,
            false
            ]
        }
    }

I saw that you used the above parameter passing method, but I used this way to report the parameter error;

I can use the following method to pass parameters, but I feel that the above method is the latest method, the following method will be deprecated;
Very confused!

"alwayson_scripts": {
    "controlnet": {
      "args": [
        {
          "input_image": "base64...",
          "model": "diff_control_sd15_depth_fp16 [978ef0a1]"
        }
      ]
    }
  }````

Vespinian · 2023-03-17T01:35:08Z

What I posted is no longer valid after the latest commits in the controlnet extension repo, The way you posted below is the right way to do it. What is being deprecated is not the method to pass the args you posted below, it's the controlnet api routes /controlnet/*2img which were used before this PR. Sorry for all the confusion, things have been moving fast.

Hugo-Matias · 2023-03-21T16:02:55Z

I've followed the recent ControlNet's api changes a bit but I'm still unsure how to proceed with other scripts. Do they need a customized handler to consume the always_on field like CN or is something provided out of the box?

DynamicPrompts and Cutoff are two interesting scripts that I'm trying to use via API, however, I don't know what to pass into the call payload. I assume that the dictionary method is a custom implementation provided by the ControlNet guys but if we can send a mixed type list how should we define what arguments to send and in what order? Maybe the process function signature is the answer?

Vespinian · 2023-03-21T17:24:57Z

Yes Controlnet made a dict out of it's args to make it more user friendly (That being said, the value of "args" is still a list of args, except each arg is a controlnetunit dict). For other scripts, it should just be the order the args are passed to the process function. So, looking at the code, I would assume for Cutoff it would look something like this.

...
"Cutoff":
{
  "args" : [bool, str, Union[float,int], bool, bool, Union[str,int], str, bool]
}
...

It should be the same list that the ui function returns L178:

        return [
            enabled,
            targets,
            weight,
            disable_neg,
            strong,
            padding,
            lerp,
            debug,
        ]

If it's not neatly laid out in the code you'll have to deduce it yourself (you could add prints and try to work it out with the ui, that's what I did with additional networks). I don't really know if there's something you can do about that at the webui's level as we only provide a reserved location in the script_arg list without really knowing what the script does with it. The API just takes the list you provided it in "args" and dumps it in the script's reserved slots in the script_args list.

Hugo-Matias · 2023-03-21T17:27:53Z

Thanks for the reply @Vespinian I was just editing my comment to say that further testing proved fruitful, I managed to make Cutoff work following my assumptions. In case anyone is interested in it as well check that discussion. Thanks a lot for the PR this is very useful stuff!

Add a way for API txt2img and img2img requests to pass args to always on scripts

dellis23 · 2023-03-28T22:08:18Z

Is there straightforward documentation for this anywhere? Or a more generic way we could expose how to use these APIs to the user? Or a better way we could expose this to the user in general?

For the life of me, I can't figure out how to 1) choose "img2img alternative test" as a script for /controlnet/img2img, and 2) how to pass arguments. I can see the shape of various answers around the internet and in this thread (something like: pass them in as positional arguments under "alwayson_scripts"), but it's unclear how to figure out what each argument is.

On a broader note: how feasible would it be to add something like a button to the UI that lets the user reproduce the payload required to reproduce the current web UI request?

Hugo-Matias · 2023-03-28T23:15:23Z

@dellis23 There are 2 types of scripts in the WebUI, the ones that stay always on and can be called upon simultaneously with other always on scripts. Not a requirement but these are the ones that usually introduce something new to the UI like ControlNet's expandable panel or a new Tab at the top.

On the other hand you have the scripts that only run when you pick them from the Scripts list on Txt2Img or Img2Img modes. The main difference is that you can only activate 1 of these scripts per generation. For example, XYZ Plot or the img2img alternative test you are trying to call.

To know if the script falls into a particular category, just open the main .py and look for the def show(): function, if it returns something like scripts.AlwaysVisible it's an always on script, otherwise, it might return something like is_img2img in that case it means that it's a regular script that is only used in Img2Img, if there is no def show() it means that it's a regular script that can be used in both modes.

Knowing this is relevant because the way you call them via the api will depend on their type. If you check the Swagger documentation page for the Txt2Img or Img2Img POST endpoints you'll notice that the payload can have both "script_name" and the new "alwayson_scripts" properties. You use "script_name" if you want to call a regular script (like you're trying to do) and "alwayson_scripts" for the other type.

Now for the arguments. There is also a "script_args" property that you must use for regular scripts. For "alwayson_scripts" the args must be sent as a dictionary with the scripts name as key. To know the script's title check the def title(): function.

The args must be sent in the order they are consumed by the webui's internal script runner, so you must check what they are and their proper order. I like to start by looking at the def process() or def run() functions, their signature is, for most cases, the argument list you need to pass (except for self and p, those are to be handled internally and you need to ignore them).

So, img2img alternative test's script file is img2imgalt.py, and is inside the scripts folder, if you open it up and scroll down you see that is a regular type of script that is shown in img2img only because the show function returns is_img2img. We should use "script_name" property in the payload and "script_args" for the arguments.

As for the arguments, this is the run function signature:

def run(self, p, _, override_sampler, override_prompt, original_prompt, original_negative_prompt, override_steps, st, override_strength, cfg, randomness, sigma_adjustment):

Like I said you must ignore self and p but if you send the other arguments in that particular order with their proper types, the chances of successfully calling the script from the api are high. Not that the first argument _ means that it is discarded by the process method but you must send a value either way, so just send some dummy data (like an empty string, 0 or false etc.) to keep the webui happy.

I'm not familiarized with that script and their types but if you compare the variable names with what is displayed in the ui you can easily reverse engineer what must be sent. Checking the def ui() is also a good way to spot what the variables need to be and what they are used for.

Here is an example payload:

{
    ...  // Img2Img info you want to send like input image as base64 string, prompt, negative prompt, seed, sampler, etc.
    "script_name": "img2img alternative test",
    "script_args": [
        0,  // _
        false,  // override_sampler: bool
        "some text",  // override_prompt: string
        "some other text",  // original_prompt: string
        "yet more text",  // original_negative_prompt: string
        true,  // override_steps: bool
        25, // st: int (value for Decode Steps)
        false,  // override_strength: bool
        7.5,  // cfg: float
        0.32,  // randomness: float
        true  // sigma_adjustment: bool
    ]
}

As I said, I never used the script and don't know what these values do but checking the ui function you can see what the script is expecting from the args list. You can use the webui to infer what they mean and apply them into your app accordingly. Sending this amount of args and in this order is the most important thing to take away from this.

If by button to send the payload you mean to test things out, you can use Swagger for that, there is an Api link in the bottom of the web ui that redirects to that page but it's in the /docs route. (http://localhost:7860/docs)

dellis23 · 2023-03-28T23:33:28Z

Thank you @Hugo-Matias, that helps a ton. I'm unfortunately not getting the same results I get when I run things in the web ui, but at least I'm a lot closer in debugging now.

dellis23 · 2023-03-28T23:35:11Z

And re:

If by button to send the payload you mean to test things out, you can use Swagger for that, there is an Api link in the bottom of the web ui that redirects to that page but it's in the /docs route. (http://localhost:7860/docs)

I mean something like: if I get the query working perfectly correctly in the web UI, it would be nice to have some way to simply get the required json payload to use for the API to recreate the exact same request (though I realize images would make that tricky). On multiple occasions I've tried to recreate a web ui result using the API and am unable to get the same results.

Hugo-Matias · 2023-03-28T23:37:20Z

Good luck, it will take some experimentation and I'm not 100% that's what you need to pass as arguments but I would start from there. Also, if you want to call ControlNet you need to pass it in as an "alwayson_script" alongside the regular script like:

{  
  ...
  "script_name": "img2img alternative test",
  "script_args": [
        ...
    ],
  "alwayson_scripts": {
    "ControlNet": {
      "args": [
        ...
      ]
    }
  }
}

I mean something like: if I get the query working perfectly correctly...

Yeah that would be a neat feature to have and would certainly make things easier but I don't think there is something like that available. However, this isn't too complicated, once you figure it out the first time, you can easily extrapolate that same workflow to other scripts.

Vespinian · 2023-03-29T00:02:23Z

A button would be nice maybe you could open an enhancement issue because I don't think many people will see this suggestion in a closed PR. An extension that can load the api param from an image might be possible too, I don't know if the extra info would have everything though.

In the mean time, if you are feeling hacky you could add prints to all the params passed in modules/img2img.py and modules/txt2img.py. That should be an almost 1 to 1 with the api parameters.

dellis23 · 2023-03-29T01:06:38Z

Thanks, created #9120

dellis23 · 2023-03-29T01:08:58Z

Also, in case anyone else is searching for information on this and ends up here, here's a payload that ultimately worked for me:

    url = "http://localhost:7860/sdapi/v1/img2img"
    body = {
        "resize_mode": 0,
        "sampler_index": "Euler",
        "sampler_name": "Euler",
        "steps": 20,
        "width": 912,
        "height": 512,
        "batch_size": 1,
        "cfg_scale": 2,
        "denoising_strength": 0.2,
        "seed": -1,
        "subseed": -1,
        "subseed_strength": 0,
        "init_images": [b64_image],
        "prompt": prompt,
        "negative_prompt": negative_prompt,
        "alwayson_scripts": {
            "ControlNet": {
                "args": [
                    {
                        "input_image": b64_controlnet_image,
                        "module": 'canny',  # aka preprocessor
                        "mask": "",
                        "model": "control_scribble-fp16 [c508311e]",
                        "weight": 1.0,
                        "resize_mode": "Scale to Fit (Inner Fit)",
                        "lowvram": False,
                        "processor_res": 512,
                        "threshold_a": 100,
                        "threshold_b": 200,
                        "guidance": 1,
                        "guidance_start": 0,
                        "guidance_end": 1,
                        "guessmode": False,
                    }
                ]
            }
        },
        "script_name": "img2img alternative test",
        "script_args": [
            None, # A dropped argument?
            True, # Override sampling method
            True, # Override prompt
            prompt,
            negative_prompt,
            True, # Override sampling steps
            50, # Decode steps
            True, # Override denoising strength
            2, # Decode CFG
            0, # Randomness
            True, # sigma adjustment
        ],
    }

LLLYF · 2023-05-08T03:53:31Z

另外，如果其他人正在搜索有关此的信息并最终到达这里，这里有一个最终对我有用的有效载荷：

    url = "http://localhost:7860/sdapi/v1/img2img"
    body = {
        "resize_mode": 0,
        "sampler_index": "Euler",
        "sampler_name": "Euler",
        "steps": 20,
        "width": 912,
        "height": 512,
        "batch_size": 1,
        "cfg_scale": 2,
        "denoising_strength": 0.2,
        "seed": -1,
        "subseed": -1,
        "subseed_strength": 0,
        "init_images": [b64_image],
        "prompt": prompt,
        "negative_prompt": negative_prompt,
        "alwayson_scripts": {
            "ControlNet": {
                "args": [
                    {
                        "input_image": b64_controlnet_image,
                        "module": 'canny',  # aka preprocessor
                        "mask": "",
                        "model": "control_scribble-fp16 [c508311e]",
                        "weight": 1.0,
                        "resize_mode": "Scale to Fit (Inner Fit)",
                        "lowvram": False,
                        "processor_res": 512,
                        "threshold_a": 100,
                        "threshold_b": 200,
                        "guidance": 1,
                        "guidance_start": 0,
                        "guidance_end": 1,
                        "guessmode": False,
                    }
                ]
            }
        },
        "script_name": "img2img alternative test",
        "script_args": [
            None, # A dropped argument?
            True, # Override sampling method
            True, # Override prompt
            prompt,
            negative_prompt,
            True, # Override sampling steps
            50, # Decode steps
            True, # Override denoising strength
            2, # Decode CFG
            0, # Randomness
            True, # sigma adjustment
        ],
    }

It was the old version about the webui API. Did you updated it and test this code could run?

Vespinian added 3 commits February 27, 2023 21:51

small refactor of api.py

a39c4cf

comment clarification

c6c2a59

Vespinian requested a review from AUTOMATIC1111 as a code owner February 28, 2023 05:19

Vespinian changed the title ~~Add an way for API request to pass args to always on scripts~~ Add a way for API request to pass args to always on scripts Feb 28, 2023

Vespinian changed the title ~~Add a way for API request to pass args to always on scripts~~ Add a way for API txt2img and img2img requests to pass args to always on scripts Feb 28, 2023

This was referenced Feb 28, 2023

Is there a way to use additional networks in the webui api? kohya-ss/sd-webui-additional-networks#105

Closed

[Feature Request]: use extensions in API call #8253

Closed

Vespinian added 5 commits March 11, 2023 12:21

Changed alwayson_script_name and alwayson_script_args api params to 1…

2174f58

… alwayson_scripts param dict

Merge branch 'master' of https://github.com/AUTOMATIC1111/stable-diff…

46f9fe3

…usion-webui

Fixed whitespace

5546e71

small diff whitespace cleanup

49bbdbe

Merge branch 'master' of https://github.com/AUTOMATIC1111/stable-diff…

64efb3d

…usion-webui

AUTOMATIC1111 approved these changes Mar 12, 2023

View reviewed changes

AUTOMATIC1111 merged commit ab0b0e1 into AUTOMATIC1111:master Mar 12, 2023

PhoenixCreation mentioned this pull request Mar 12, 2023

[API-UPDATE] Need to update to accommodate recent API changes in AUTOMATIC1111 Mikubill/sd-webui-controlnet#567

Closed

3 tasks

xwwxyd mentioned this pull request Mar 14, 2023

when call the api, I got TypeError: 'NoneType' object is not subscriptable Mikubill/sd-webui-controlnet#578

Closed

serg4kostiuk pushed a commit to talkable/stable-diffusion-webui that referenced this pull request Mar 22, 2023

Merge pull request AUTOMATIC1111#8187 from Vespinian/master

d5741b8

Add a way for API txt2img and img2img requests to pass args to always on scripts

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a way for API txt2img and img2img requests to pass args to always on scripts #8187

Add a way for API txt2img and img2img requests to pass args to always on scripts #8187

Vespinian commented Feb 28, 2023 •

edited

Loading

AUTOMATIC1111 commented Mar 11, 2023

Vespinian commented Mar 11, 2023

mengchengwanli commented Mar 16, 2023

Vespinian commented Mar 16, 2023

mengchengwanli commented Mar 16, 2023

Vespinian commented Mar 16, 2023

mengchengwanli commented Mar 17, 2023

Vespinian commented Mar 17, 2023 •

edited

Loading

Hugo-Matias commented Mar 21, 2023

Vespinian commented Mar 21, 2023 •

edited

Loading

Hugo-Matias commented Mar 21, 2023

dellis23 commented Mar 28, 2023

Hugo-Matias commented Mar 28, 2023 •

edited

Loading

dellis23 commented Mar 28, 2023

dellis23 commented Mar 28, 2023

Hugo-Matias commented Mar 28, 2023 •

edited

Loading

Vespinian commented Mar 29, 2023

dellis23 commented Mar 29, 2023

dellis23 commented Mar 29, 2023

LLLYF commented May 8, 2023

Add a way for API txt2img and img2img requests to pass args to always on scripts #8187

Add a way for API txt2img and img2img requests to pass args to always on scripts #8187

Conversation

Vespinian commented Feb 28, 2023 • edited Loading

AUTOMATIC1111 commented Mar 11, 2023

Vespinian commented Mar 11, 2023

mengchengwanli commented Mar 16, 2023

Vespinian commented Mar 16, 2023

mengchengwanli commented Mar 16, 2023

Vespinian commented Mar 16, 2023

mengchengwanli commented Mar 17, 2023

Vespinian commented Mar 17, 2023 • edited Loading

Hugo-Matias commented Mar 21, 2023

Vespinian commented Mar 21, 2023 • edited Loading

Hugo-Matias commented Mar 21, 2023

dellis23 commented Mar 28, 2023

Hugo-Matias commented Mar 28, 2023 • edited Loading

dellis23 commented Mar 28, 2023

dellis23 commented Mar 28, 2023

Hugo-Matias commented Mar 28, 2023 • edited Loading

Vespinian commented Mar 29, 2023

dellis23 commented Mar 29, 2023

dellis23 commented Mar 29, 2023

LLLYF commented May 8, 2023

Vespinian commented Feb 28, 2023 •

edited

Loading

Vespinian commented Mar 17, 2023 •

edited

Loading

Vespinian commented Mar 21, 2023 •

edited

Loading

Hugo-Matias commented Mar 28, 2023 •

edited

Loading

Hugo-Matias commented Mar 28, 2023 •

edited

Loading