Skip to content

API

Jump to bottom
ljleb edited this page Mar 13, 2023 · 37 revisions

The extension has 2 APIs:

  • external code API
  • web API

The external code API is useful when you want to control this extension from another extension.

The web API is useful when you want to communicate with the extension from a web client.

External Code API

The extension defines the external_code module. This module contains functions that you can use to control the extension for generating.

Examples

Create ControlNet Arguments

To create custom arguments and pass them to the extension for generating:

import importlib
external_code = importlib.import_module('extensions.sd-webui-controlnet.scripts.external_code', 'external_code')

def create_script_args(p: StableDiffusionProcessing):
    models = external_code.get_models()
    cn_units = [
        external_code.ControlNetUnit(
            model=models[0],  # assuming at least 1 model exists
            ...
        ),
        external_code.ControlNetUnit(
            model=models[1],  # assuming at least 2 models exist
            ...
        ),
        ...
    ]
    external_code.update_cn_script_in_processing(p, cn_units)

Update Existing Arguments

To update the ControlNet processing units from an existing script runner:

import importlib
external_code = importlib.import_module('extensions.sd-webui-controlnet.scripts.external_code', 'external_code')

def update_script_args(p: StableDiffusionProcessing):
    cn_units = external_code.get_all_units_in_processing(p)
    cn_units[0].resize_mode = external_code.ResizeMode.RESIZE
    cn_units[0].model = ...
    cn_units[1].image = None
    ...
    external_code.update_cn_script_in_processing(p, updated_units)

Remove All ControlNet Arguments

To remove all ControlNet processing units from an existing script runner: (essentially has the same effect as disabling the extension)

import importlib
external_code = importlib.import_module('extensions.sd-webui-controlnet.scripts.external_code', 'external_code')

def disable_controlnet(p: StableDiffusionProcessing):
    external_code.update_cn_script_in_processing(p, [])

Web API

API Update: The /controlnet/txt2img and /controlnet/img2img routes are deprecated. Consider migrating to the /sdapi/v1/txt2img and /sdapi/v1/img2img routes instead.

The extension adds the following routes to the web API of the webui:

All routes use the Content-Type: application/json header.

Route POST /controlnet/txt2img

This route can be used exactly like /sdapi/v1/txt2img. It accepts the same json properties, with the addition of an extra property "controlnet_units": []. This property accepts a list of ControlNetUnitRequest objects. Here's an example request body:

{
  "prompt": "a cinematic shot of an impressive ants war, ant melee, armageddon",
  "sampler_name": "Euler",
  "controlnet_units": [
    {
      "input_image": "base64...",
      "model": "diff_control_sd15_depth_fp16 [978ef0a1]"
    }
  ]
}

Route POST /controlnet/img2img

This route can be used exactly like /sdapi/v1/img2img. It accepts the same json properties, with the addition of an extra property "controlnet_units": []. This property accepts a list of ControlNetUnitRequest objects. Here's an example request body:

{
  "init_images": ["base64..."],
  "sampler_name": "Euler",
  "controlnet_units": [
    {
      "module": "depth",
      "model": "diff_control_sd15_depth_fp16 [978ef0a1]"
    }
  ]
}

Route GET /controlnet/model_list

Get the list of available ControlNet models. Returns a dictionary of the form {"model_list": [...]}. Each value of "model_list" is a valid candidate for the "model" property of the ControlNetUnitRequest object described below.

Route POST /controlnet/detect

Run a preprocessor by itself. Body of the route accepts a JSON object with the following property:

  • "controlnet_module" : preprocessor to use. defaults to "None"
  • "controlnet_input_images" : images to process. defaults to []
  • "controlnet_processor_res" : resolution of the preprocessor. defaults to 512
  • "controlnet_threshold_a" : first parameter of the preprocessor. only takes effect when preprocessor accepts arguments. defaults to 64
  • "controlnet_threshold_b" : second parameter of the preprocessor, same as "controlnet_threshold_a" for usage. defaults to 64

ControlNetUnitRequest JSON Object

This object describes a ControlNet processing unit entirely. It has the following properties:

  • "input_image" : image to use in this unit. defaults to ""
  • "module" : preprocessor to use on the image passed to this unit before using it for conditioning. defaults to "none"
  • "model" : name of the model to use for conditioning in this unit. accepts values returned by /controlnet/model_list route. defaults to "None"
  • "weight" : weight of this unit. defaults to 1
  • "resize_mode" : how to resize the input image so as to fit the output resolution of the generation. defaults to "Scale to Fit (Inner Fit)". Accepted values:
    • 0 or "Just Resize" : simply resize the image to the target width/height
    • 1 or "Scale to Fit (Inner Fit)" : scale and crop to fit smallest dimension. preserves proportions.
    • 2 or "Envelope (Outer Fit)" : scale to fit largest dimension. preserves proportions.
  • "lowvram" : whether to compensate low GPU memory with processing time. defaults to false
  • "processor_res" : resolution of the preprocessor. defaults to 64
  • "threshold_a" : first parameter of the preprocessor. only takes effect when preprocessor accepts arguments. defaults to 64
  • "threshold_b" : second parameter of the preprocessor, same as above for usage. defaults to 64
  • "guidance_start" : ratio of generation where this unit starts to have an effect. defaults to 0.0
  • "guidance_end" : ratio of generation where this unit stops to have an effect. defaults to 1.0
  • "guidance" : deprecated. same as "guidance_end" above. defaults to 1
  • "guessmode" : whether to infer and append a basic prompt for generation. defaults to true

Migrating from /controlnet/*2img to /sdapi/v1/*2img

Disclaimer: this section is subject to change rapidly in the next days. The target structure should stay similar overall but there may be 1 or 2 minor changes in the final revision.

ControlNetUnitRequest objects do not have to be updated. ControlNetUnitRequest json objects can just be forwarded to the argument list of the ControlNet script, with the addition of 1 boolean value representing whether the extension is to pretend to run from the gradio interface. (named is_ui in code)

Examples

This is the /controlnet/txt2img example converted to work with /sdapi/v1/txt2img:

{
  "prompt": "a cinematic shot of an impressive ants war, ant melee, armageddon",
  "sampler_name": "Euler",
  "alwayson_scripts": {
    "controlnet": {
      "args": [
        false,
        {
          "input_image": "base64...",
          "model": "diff_control_sd15_depth_fp16 [978ef0a1]"
        }
      ]
    }
  }
}

The boolean value false before controlnet objects tells controlnet that it has been called from the API.

This is the /controlnet/img2img example converted to work with /sdapi/v1/img2img:

{
  "init_images": ["base64..."],
  "sampler_name": "Euler",
  "alwayson_scripts": {
    "controlnet": {
      "args": [
        false,
        {
          "module": "depth",
          "model": "diff_control_sd15_depth_fp16 [978ef0a1]"
        }
      ]
    }
  }
}

The boolean value false before controlnet objects tells controlnet that it has been called from the API.

Clone this wiki locally