Skip to content

API

Jump to bottom
Chenlei Hu edited this page Jul 6, 2024 · 37 revisions

The extension has 2 APIs:

  • external code API
  • web API

The external code API is useful when you want to control this extension from another extension.

The web API is useful when you want to communicate with the extension from a web client.

External Code API

The extension defines the external_code module. This module contains functions that you can use to control the extension for generating.

Examples

Create ControlNet Arguments

To create custom arguments and pass them to the extension for generating:

import importlib
external_code = importlib.import_module('extensions.sd-webui-controlnet.scripts.external_code', 'external_code')

def create_script_args(p: StableDiffusionProcessing):
    models = external_code.get_models()
    cn_units = [
        external_code.ControlNetUnit(
            model=models[0],  # assuming at least 1 model exists
            ...
        ),
        external_code.ControlNetUnit(
            model=models[1],  # assuming at least 2 models exist
            ...
        ),
        ...
    ]
    external_code.update_cn_script_in_processing(p, cn_units)

Update Existing Arguments

To update the ControlNet processing units from an existing script runner:

import importlib
external_code = importlib.import_module('extensions.sd-webui-controlnet.scripts.external_code', 'external_code')

def update_script_args(p: StableDiffusionProcessing):
    cn_units = external_code.get_all_units_in_processing(p)
    cn_units[0].resize_mode = external_code.ResizeMode.RESIZE
    cn_units[0].model = ...
    cn_units[1].image = None
    ...
    external_code.update_cn_script_in_processing(p, updated_units)

Remove All ControlNet Arguments

To remove all ControlNet processing units from an existing script runner: (essentially has the same effect as disabling the extension)

import importlib
external_code = importlib.import_module('extensions.sd-webui-controlnet.scripts.external_code', 'external_code')

def disable_controlnet(p: StableDiffusionProcessing):
    external_code.update_cn_script_in_processing(p, [])

Web API

API Update: The /controlnet/txt2img and /controlnet/img2img routes have been removed. Please use the /sdapi/v1/txt2img and /sdapi/v1/img2img routes instead.

The extension adds the following routes to the web API of the webui:

All routes use the Content-Type: application/json header.

Route GET /controlnet/model_list

Get the list of available ControlNet models. Returns a dictionary of the form {"model_list": [...]}. Each value of "model_list" is a valid candidate for the "model" property of the ControlNetUnitRequest object described below.

Route GET /controlnet/module_list

Get the list of available preprocessors. Returns a dictionary of the form {"module_list": [...]}. Each value of "module_list" is a valid candidate for the "module" property of the ControlNetUnitRequest object described below.

Request parameters:

  • alias_names=true : whether to get the ui alias names instead of internal keys. Defaults to false

Route POST /controlnet/detect

Run a preprocessor by itself. Body of the route accepts a JSON object with the following property:

  • "controlnet_module" : preprocessor to use. defaults to "none"
  • "controlnet_input_images" : images to process. defaults to []
  • "controlnet_processor_res" : resolution of the preprocessor. defaults to 512
  • "controlnet_threshold_a" : first parameter of the preprocessor. only takes effect when preprocessor accepts arguments. defaults to 64
  • "controlnet_threshold_b" : second parameter of the preprocessor, same as "controlnet_threshold_a" for usage. defaults to 64

Route GET /controlnet/version

Get running API version. Returns a dictionary of the form {"version": n} where n is an integer.

The current API version is 1.

ControlNetUnitRequest JSON Object

This object describes a ControlNet processing unit entirely. It has the following properties:

  • "enabled" : Whether this unit is enabled. defaults to False
  • "image" : image to use in this unit. defaults to null
  • "mask" : mask pixel_perfect to filter the image. defaults to null
  • "module" : preprocessor to use on the image passed to this unit before using it for conditioning. accepts values returned by the /controlnet/module_list route. defaults to "none"
  • "model" : name of the model to use for conditioning in this unit. accepts values returned by the /controlnet/model_list route. defaults to "None"
  • "weight" : weight of this unit. defaults to 1
  • "resize_mode" : how to resize the input image so as to fit the output resolution of the generation. defaults to "Scale to Fit (Inner Fit)". Accepted values:
    • 0 or "Just Resize" : simply resize the image to the target width/height
    • 1 or "Scale to Fit (Inner Fit)" : scale and crop to fit smallest dimension. preserves proportions.
    • 2 or "Envelope (Outer Fit)" : scale to fit largest dimension. preserves proportions.
  • "lowvram" : whether to compensate low GPU memory with processing time. defaults to false
  • "processor_res" : resolution of the preprocessor. defaults to 64
  • "threshold_a" : first parameter of the preprocessor. only takes effect when preprocessor accepts arguments. defaults to 64
  • "threshold_b" : second parameter of the preprocessor, same as above for usage. defaults to 64
  • "guidance_start" : ratio of generation where this unit starts to have an effect. defaults to 0.0
  • "guidance_end" : ratio of generation where this unit stops to have an effect. defaults to 1.0
  • "control_mode" : see the related issue for usage. defaults to 0. Accepted values:
    • 0 or "Balanced" : balanced, no preference between prompt and control model
    • 1 or "My prompt is more important" : the prompt has more impact than the model
    • 2 or "ControlNet is more important" : the controlnet model has more impact than the prompt
  • "pixel_perfect" : enable pixel-perfect preprocessor. defaults to false

Integrating /sdapi/v1/*2img

Pass ControlNetUnitRequest objects to the argument list of the ControlNet script.

Examples

This is the example to work with /sdapi/v1/txt2img:

{
  "prompt": "a cinematic shot of an impressive ants war, ant melee, armageddon",
  "sampler_name": "Euler",
  "alwayson_scripts": {
    "controlnet": {
      "args": [
        {
          "enabled": true,
          "image": "base64...",
          "model": "diff_control_sd15_depth_fp16 [978ef0a1]"
        }
      ]
    }
  }
}

This is the example to work with /sdapi/v1/img2img:

{
  "init_images": ["base64..."],
  "sampler_name": "Euler",
  "alwayson_scripts": {
    "controlnet": {
      "args": [
        {
          "enabled": true,
          "module": "depth",
          "model": "diff_control_sd15_depth_fp16 [978ef0a1]"
        }
      ]
    }
  }
}

Below is a minimal working example for sanity check (this example is tested every day and you can trust that it should always work.)

import io
import cv2
import base64
import requests


from PIL import Image


# A1111 URL
url = "http://127.0.0.1:7860"

# Read Image in RGB order
img = cv2.imread('your_image.jpg')

# Encode into PNG and send to ControlNet
retval, bytes = cv2.imencode('.png', img)
encoded_image = base64.b64encode(bytes).decode('utf-8')

# A1111 payload
payload = {
    "prompt": 'a handsome man',
    "negative_prompt": "",
    "batch_size": 1,
    "steps": 20,
    "cfg_scale": 7,
    "alwayson_scripts": {
        "controlnet": {
            "args": [
                {
                    "enabled": True,
                    "image": encoded_image,
                    "module": "canny",
                    "model": "control_v11p_sd15_canny [d14c016b]",
                }
            ]
        }
    }
}


# Trigger Generation
response = requests.post(url=f'{url}/sdapi/v1/txt2img', json=payload)

# Read results
r = response.json()
result = r['images'][0]
image = Image.open(io.BytesIO(base64.b64decode(result.split(",", 1)[0])))
image.save('output.png')

See also WebUI's sd-api-v1.

Clone this wiki locally