Support loading quantized models from quanto directly

[sayakpaul]

With more and more and larger and larger diffusion transformer models coming up, I think it makes sense to support the direct loading of quantized models from quanto.

Currently, the workflow to load a quantized model with quanto is simple:

from diffusers import DiffusionPipeline
from optimum.quanto import QuantizedPixArtTransformer2DModel
import torch

transformer = QuantizedPixArtTransformer2DModel.from_pretrained("pixart-sigma-fp8").to("cuda", torch.float16) 
pipe = DiffusionPipeline.from_pretrained(
    "PixArt-alpha/PixArt-Sigma-XL-2-1024-MS", 
    transformer=None,
    torch_dtype=torch.float16,
).to("cuda")
pipe.transformer = transformer

We should be able to just do:

pipe = PixArtSigmaPipeline.from_pretrained(
    "PixArt-alpha/PixArt-Sigma-XL-2-1024-MS", 
-    transformer=None,
+    transformer=transformer,
    torch_dtype=torch.float16,
).to("cuda")

Currently, transformer=transformer would fail because there's no mapping here:

diffusers/src/diffusers/pipelines/pipeline_loading_utils.py

Line 67 in b1f43d7

LOADABLE_CLASSES = {

Thoughts @yiyixuxu @DN6?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[sayakpaul]

Closing this issue because we will add quanto quantization backend after #9213 is done.