AutoPipeline.from_pipe fails when passed a ControlNet pipeline

[carson-katri]

Describe the bug

The implementation of AutoPipeline does a replace of "Pipeline" with "ControlNetPipeline" when the controlnet argument is passed.

When from_pipe is called with an existing ControlNet pipeline, the following transformation occurs.

StableDiffusionControlNetPipeline --> StableDiffusionControlNetControlNetPipeline

This results in an error, as the StableDiffusionControlNetControlNetPipeline class does not exist.

Reproduction

from diffusers import AutoPipelineForText2Image, ControlNetModel

canny = ControlNetModel.from_pretrained("lllyasviel/control_v11p_sd15_canny")

# automatically chooses `StableDiffusionControlNetPipeline`
controlnet_pipe1 = AutoPipelineForText2Image.from_pretrained("runwayml/stable-diffusion-v1-5", controlnet=canny)

# replaces `Pipeline` with `ControlNetPipeline` -> `StableDiffusionControlNetControlNetPipeline`
controlnet_pipe2 = AutoPipelineForText2Image.from_pipe(controlnet_pipe1, controlnet=canny)
# ValueError: AutoPipeline can't find a pipeline linked to StableDiffusionControlNetControlNetPipeline for None

Logs

ValueError: AutoPipeline can't find a pipeline linked to StableDiffusionControlNetControlNetPipeline for None

System Info

• diffusers version: 0.21.4
• Platform: macOS-14.1-arm64-arm-64bit
• Python version: 3.10.6
• PyTorch version (GPU?): 2.0.1 (False)
• Huggingface_hub version: 0.16.4
• Transformers version: 4.30.2
• Accelerate version: 0.18.0
• xFormers version: 0.0.15.dev+71205ec.d20221126
• Using GPU in script?: no
• Using distributed or parallel set-up in script?: no

Who can help?

@yiyixuxu

[patrickvonplaten]

@yiyixuxu could you take a look here?

[carson-katri]

Working great now, thanks!

[CyrusVorwald]

This should be reopened because it still fails for StableDiffusionXLControlNetInpaintPipeline. I have opened a PR here #6302