Add inpainting pipeline to diffuser conversion

[HoopyFreud]

Currently trying to add an inpainting model results in a size mistmatch error during conversion. Desired bhavior is to convert the inpainting model.

Problem appears to be in https://github.com/ssube/onnx-web/blob/main/api/onnx_web/convert/diffusion/diffusers.py#L3

line 214, there should be an alternate pipeline (https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_inpaint.py) for inpainting diffusers, and around line 421 as well (https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_onnx_stable_diffusion_inpaint.py)

[HoopyFreud]

If we want to implement an inpainting model type for the extras file, https://github.com/ssube/onnx-web/blob/main/api/onnx_web/convert/__main__.py will also need to be changed around line 339

[ssube]

I have some code for this on the same branch as tiled VAE. If I can get that working today, I'll merge both, otherwise I will cherry-pick this fix.

[ssube]

I added support for the inpaint pipeline to the diffusers (folder) conversion code, but it's still getting stuck in the from-checkpoint code:

[2023-04-29 15:18:29,901] INFO: 3115610 140346653454336 onnx_web.convert.diffusion.original: Pred and size are epsilon and 512, using config: /opt/onnx-web/models/configs/v1-training-default.yaml                 
[2023-04-29 15:18:29,901] INFO: 3115610 140346653454336 onnx_web.convert.diffusion.original: v1 model loaded.                                                                                                       
[2023-04-29 15:18:29,901] DEBUG: 3115610 140346653454336 onnx_web.convert.diffusion.original: trying to load: /opt/onnx-web/models/configs/v1-training-default.yaml                                                 
[2023-04-29 15:18:29,906] INFO: 3115610 140346653454336 onnx_web.convert.diffusion.original: converting UNet                                                                                                        
[2023-04-29 15:18:33,905] ERROR: 3115610 140346653454336 onnx_web.convert.diffusion.original: error setting up output                                                                                               
Traceback (most recent call last):                                                                                                                                                                                  
  File "/opt/onnx-web/api/onnx_web/convert/diffusion/original.py", line 1526, in extract_checkpoint
    unet.load_state_dict(converted_unet_checkpoint)
  File "/opt/onnx-web/api/onnx_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1671, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for UNet2DConditionModel:
        size mismatch for conv_in.weight: copying a param with shape torch.Size([320, 9, 3, 3]) from checkpoint, the shape in current model is torch.Size([320, 4, 3, 3]).

I need to figure out if that should be fixed in place or if #337 will fix this.

[ssube]

The .from_ckpt method available in diffusers seems to have the same shape problem:

Traceback (most recent call last):
  File "/opt/onnx-web/api/onnx_web/convert/__main__.py", line 352, in convert_models
    converted, dest = convert_diffusion_diffusers(
  File "/opt/onnx-web/api/onnx_env/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/opt/onnx-web/api/onnx_web/convert/diffusion/diffusers.py", line 242, in convert_diffusion_diffusers
    pipeline = pipe_class.from_ckpt(
  File "/opt/onnx-web/api/onnx_env/lib/python3.10/site-packages/diffusers/loaders.py", line 1284, in from_ckpt
    pipe = download_from_original_stable_diffusion_ckpt(
  File "/opt/onnx-web/api/onnx_env/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion/convert_from_ckpt.py", line 1181, in download_from_original_stable_diffusion_ckpt
    unet.load_state_dict(converted_unet_checkpoint)
  File "/opt/onnx-web/api/onnx_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1671, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for UNet2DConditionModel:
        size mismatch for conv_in.weight: copying a param with shape torch.Size([320, 9, 3, 3]) from checkpoint, the shape in current model is torch.Size([320, 4, 3, 3]).

The stock SD v1.5 and v2.1 inpainting models convert correctly. This one appears to have a different shape, but that same shape error appears if you use one of the successfully-converted inpainting models with a non-inpainting pipeline, so it's not totally new.

I need to compare the tensors in the models and see what is actually different here. It may still be a bug in the ckpt loader.

[ssube]

Per huggingface/diffusers#3254 (comment), num_in_channels needs to be set in the call to .from_ckpt and underlying call to download_from_original_stable_diffusion_ckpt.

It seems like the most efficient way to do this will be to create the new UNet2DConditionModel_CNet using .from_config(unet.config) and the original pipeline's UNet instance, which has already been loaded for both full conversions and CNet-only.