[Flux] Optimize guidance creation in flux pipeline by moving it outsi… #3654

Job	Run time
Setup Torch Pipelines CUDA Slow Tests Matrix	2m 25s
Torch CUDA Tests (models)	4m 47s
Torch CUDA Tests (schedulers)	2m 59s
Flax TPU Tests	1m 53s
Torch CUDA Tests (lora)	5m 25s
Torch CUDA Tests (others)	2m 9s
ONNX CUDA Tests	7m 13s
Torch CUDA Tests (single_file)	4m 16s
Examples PyTorch CUDA tests on Ubuntu	22m 19s
PyTorch xformers CUDA tests	4m 51s
PyTorch Compile CUDA tests	2m 45s
Torch Pipelines CUDA Tests (controlnet)	9m 44s
Torch Pipelines CUDA Tests (ddpm)	2m 0s
Torch Pipelines CUDA Tests (deepfloyd_if)	3m 35s
Torch Pipelines CUDA Tests (flux)	2m 10s
Torch Pipelines CUDA Tests (ip_adapters)	2m 1s
Torch Pipelines CUDA Tests (kandinsky)	4m 12s
Torch Pipelines CUDA Tests (kandinsky2_2)	4m 30s
Torch Pipelines CUDA Tests (latent_consistency_models)	2m 43s
Torch Pipelines CUDA Tests (pixart_alpha)	2m 14s
Torch Pipelines CUDA Tests (stable_cascade)	2m 46s
Torch Pipelines CUDA Tests (stable_diffusion)	4m 35s
Torch Pipelines CUDA Tests (stable_diffusion_2)	5m 34s
Torch Pipelines CUDA Tests (stable_diffusion_3)	2m 40s
Torch Pipelines CUDA Tests (stable_diffusion_adapter)	3m 7s
Torch Pipelines CUDA Tests (stable_diffusion_xl)	5m 51s
Torch Pipelines CUDA Tests (stable_video_diffusion)	2m 18s
Torch Pipelines CUDA Tests (text_to_video_synthesis)	3m 33s
Torch Pipelines CUDA Tests (wuerstchen)	2m 45s
	2h 7m 20s

Provide feedback