Flux inference og

[echosyy]

Describe the bug

I use the optimization.quanto package to call the quantization function. When the model are quantized to fp8, the speed is much slower than bf16. want to know why, thank you?

Reproduction

`transformer = FluxTransformer2DModel.from_single_file("https://huggingface.co/Kijai/flux-fp8/blob/main/flux1-dev-fp8.safetensors", torch_dtype=dtype)
quantize(transformer, weights=qfloat8)
freeze(transformer)

text_encoder_2 = T5EncoderModel.from_pretrained(bfl_repo, subfolder="text_encoder_2", torch_dtype=dtype)
quantize(text_encoder_2, weights=qfloat8)
freeze(text_encoder_2)

pipe = FluxPipeline.from_pretrained(bfl_repo, transformer=None, text_encoder_2=None, torch_dtype=dtype)
pipe.transformer = transformer
pipe.text_encoder_2 = text_encoder_2`

Logs

No response

System Info

X86, torch2.4+cuda12.2

Who can help?

No response

[asomoza]

how much slower? I think this is kind of expected but ccing @sayakpaul for more insights.

[echosyy]

bf16 is 1.8iter/s, but fp8 is 1.1iter/s,
device: L40s

[sayakpaul]

Are you not popping your model to GPU? I don't see any placements.

[echosyy]

yes, ⬇️ is the complete code.

`transformer = FluxTransformer2DModel.from_single_file("https://huggingface.co/Kijai/flux-fp8/blob/main/flux1-dev-fp8.safetensors", torch_dtype=dtype)
quantize(transformer, weights=qfloat8)
freeze(transformer)

transformer = torch.ao.quantization.quantize_dynamic(
transformer, # the original model
{torch.nn.Linear}, # a set of layers to dynamically quantize
dtype=torch.qint8) # the target dtype for quantized weights

text_encoder_2 = T5EncoderModel.from_pretrained(bfl_repo, subfolder="text_encoder_2", torch_dtype=dtype)
quantize(text_encoder_2, weights=qfloat8)
freeze(text_encoder_2)

pipe = FluxPipeline.from_pretrained(bfl_repo, transformer=None, text_encoder_2=None, torch_dtype=dtype)
pipe.transformer = transformer
pipe.text_encoder_2 = text_encoder_2

pipe = pipe.to('cuda')
prompt = "A cat holding a sign that says hello world"

image = pipe(
prompt,
height=1024,
width=1024,
guidance_scale=3.5,
num_inference_steps=20,
max_sequence_length=512,
generator=torch.Generator("cuda").manual_seed(0)).images[0]`

[sayakpaul]

Also, going to move this to discussions as this is not a library issue.