Visual artifacts when using DPM++ schedulers and SDXL without the refiner model

[CodeCorrupt]

Describe the bug

All DPM++ schedulers are showing visual artifacts out of the base model when denoising_end=1 (skipping the refiner). This effect is most notable with DPM++ 2M SDE configured using the flag from the docs.

These same artifacts are not seen when using SD1.5 with the same scheduler configuration.

Reproduction

Intended to run in a notebook

import torch
from diffusers import StableDiffusionXLPipeline, StableDiffusionPipeline
from typing import cast
from diffusers import DPMSolverMultistepScheduler

sdxl_model = cast(StableDiffusionXLPipeline, StableDiffusionXLPipeline.from_pretrained(
    'stabilityai/stable-diffusion-xl-base-1.0',
    torch_dtype=torch.float16,
    use_safetensors=True,
    variant="fp16",
)).to('cuda')
sd_model = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16,
    revision="fp16",
).to('cuda')

common_config = {'beta_start': 0.00085, 'beta_end': 0.012, 'beta_schedule': 'scaled_linear'}
dpmpp_2m_sde = DPMSolverMultistepScheduler(**common_config, algorithm_type="sde-dpmsolver++")
sdxl_model.scheduler = dpmpp_2m_sde
sd_model.scheduler = dpmpp_2m_sde

sdxl_model.watermark = None
generator = torch.Generator(device='cuda')
generator.manual_seed(12345)

params = {
    "prompt": ['a cat'],
    "num_inference_steps": 50,
    "height": 1024,
    "width": 1024,
    "guidance_scale": 7,
}

sdxl_res = sdxl_model(**params, denoising_end=1.0, generator=generator)
sdxl_img = sdxl_res.images[0]

generator.manual_seed(12345)
sd_res = sd_model(**params, generator=generator)
sd_img = sd_res.images[0]

display(sdxl_img)
display(sd_img)

Logs

No response

System Info

• diffusers version: 0.21.4
• Platform: Linux-5.4.0-163-generic-x86_64-with-glibc2.31
• Python version: 3.11.5
• PyTorch version (GPU?): 2.1.0+cu121 (True)
• Huggingface_hub version: 0.17.1
• Transformers version: 4.34.0
• Accelerate version: 0.22.0
• xFormers version: not installed
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

@yiyixuxu @patrickvonplaten

[AmericanPresidentJimmyCarter]

Would be good to get this one fixed, as it's been a problem since the SDXL launch.

[yiyixuxu]

@CodeCorrupt

could you try use this script instead? a few notes:

1. you do not need to pass denoising_end=1 if you want to skip refiner. Just leave it as its default value None- passingdenoising_end=1 is actually a no-op here
   

   diffusers/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py

   Line 905 in e516858

   if denoising_end is not None and isinstance(denoising_end, float) and denoising_end > 0 and denoising_end < 1:

2. we would want to use the same config as default scheduler when swapping out for a different scheduler - in our case, the default config is a little bit different from the common_config you created, mainly step_offset and timeatep_spaing

I generated a few images, they looks fine to me but I might have less trained eyes. so let me know if the problem still persist

import torch
from diffusers import StableDiffusionXLPipeline, StableDiffusionPipeline
from typing import cast
from diffusers import DPMSolverMultistepScheduler

pipe = StableDiffusionXLPipeline.from_pretrained(
    'stabilityai/stable-diffusion-xl-base-1.0',
    torch_dtype=torch.float16,
    use_safetensors=True,
    variant="fp16",
    add_watermarker=False)
pipe = pipe.to('cuda')

pipe.scheduler = DPMSolverMultistepScheduler.from_config(
    pipe.scheduler.config, algorithm_type="sde-dpmsolver++"
)

generator = torch.Generator(device='cuda').manual_seed(12345)

params = {
    "prompt": ['a cat'],
    "num_inference_steps": 50,
    "guidance_scale": 7,
}

sdxl_img = pipe(**params, generator=generator).images[0]
sdxl_img.save(f"sdxl_dpm_out.png")

[yiyixuxu]

@CodeCorrupt
actually I think I still see them with the script I just gave you .....:cold_face:

can you try this? I think they are gone for real this time but let me know if it's not......

import torch
from diffusers import StableDiffusionXLPipeline, StableDiffusionPipeline
from typing import cast
from diffusers import DPMSolverMultistepScheduler

pipe = StableDiffusionXLPipeline.from_pretrained(
    'stabilityai/stable-diffusion-xl-base-1.0',
    torch_dtype=torch.float16,
    use_safetensors=True,
    variant="fp16",
    add_watermarker=False)
pipe = pipe.to('cuda')

pipe.scheduler  = DPMSolverMultistepScheduler.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    subfolder="scheduler")


seed = 1
generator = torch.Generator(device='cuda').manual_seed(seed)

params = {
    "prompt": ['a cat'],
    "num_inference_steps": 50,
    "guidance_scale": 7,
}

sdxl_img = pipe(**params, generator=generator).images[0]
sdxl_img.save(f"out_{seed}.png")

[CodeCorrupt]

Hey @yiyixuxu Looks like it's not using the right algorithm_type in the most recent example. I made a test script to compare.

import torch
from diffusers import StableDiffusionXLPipeline
from diffusers import DPMSolverMultistepScheduler, DPMSolverSinglestepScheduler

pipe = StableDiffusionXLPipeline.from_pretrained(
    'stabilityai/stable-diffusion-xl-base-1.0',
    torch_dtype=torch.float16,
    use_safetensors=True,
    variant="fp16",
    add_watermarker=False)
pipe = pipe.to('cuda')

common_config = {'beta_start': 0.00085, 'beta_end': 0.012, 'beta_schedule': 'scaled_linear'}
schedulers = {
    "DPMPP_2M": (DPMSolverMultistepScheduler, {}),
    "DPMPP_2M_K": (DPMSolverMultistepScheduler, {"use_karras_sigmas": True}),
    "DPMPP_2M_SDE": (DPMSolverMultistepScheduler, {"algorithm_type": "sde-dpmsolver++"}),
    "DPMPP_2M_SDE_K": (DPMSolverMultistepScheduler, {"use_karras_sigmas": True, "algorithm_type": "sde-dpmsolver++"}),
    "DPMPP_SDE": (DPMSolverSinglestepScheduler, {}),
    "DPMPP_SDE_K": (DPMSolverSinglestepScheduler, {"use_karras_sigmas": True}),
}

selected_scheduler = 'DPMPP_2M_SDE'
scheduler_old = schedulers[selected_scheduler][0](**common_config, **schedulers[selected_scheduler][1])
scheduler_new = schedulers[selected_scheduler][0].from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    subfolder="scheduler",
    **schedulers[selected_scheduler][1],
)

params = {
    "prompt": ['a cat'],
    "num_inference_steps": 50,
    "guidance_scale": 7,
}
for s in [scheduler_new, scheduler_old]:
    seed = 12345
    generator = torch.Generator(device='cuda').manual_seed(seed)

    pipe.scheduler = s
    sdxl_img = pipe(**params, generator=generator).images[0]
    display(sdxl_img)

I'm still seeing the same artifacts when using the .from_pretrained()

[yiyixuxu]

@CodeCorrupt
ahhh really sorry you're absolutely right - my results was actually based on dpmsolver +++ 🥺

@LuChengTHU can you aso take a look into this? I can reproduce this bug

[yiyixuxu]

@CodeCorrupt
actually i saw same artifacts in k-diffusion too (with auto1111)

here is my setting - do you see same thing in automatic1111 as well?

[yiyixuxu]

@CodeCorrupt
btw the artifacts would go away if you increase the number of inference steps

below, num_inference_steps = 60,70,80,100. It gradually reduce and I think completely disappeared at 100 steps

Since the same issue also present in k-diffusion/auto1111, and also because of the the fact that it went away when we increase the number of inference steps, I think this is probably not a bug in the implementation. It could be just this scheduler does not work well with SDXL. I would be curious to understand why and hopefully @LuChengTHU have some insights to share soon :) but I think there is not much action for us to take here

also cc @patrickvonplaten here, let me know if we should investigate this further

[sayakpaul]

Have we confirmed that swapping the default VAE to use this one doesn't help at all?

[yiyixuxu]

@sayakpaul

Have we confirmed that swapping the default VAE to use this one doesn't help at all?

confirmed

[patrickvonplaten]

DPM++ scheduler is known to not work super well for SDXL. Euler is usually the better choice

[yiyixuxu]

@CodeCorrupt

let us know if there is anything you want us to investigate more:)

[AmericanPresidentJimmyCarter]

DPM++ scheduler is known to not work super well for SDXL. Euler is usually the better choice

DPM++ cleans up on FID when comparing it to other methods like Euler for the same number of steps. It would be nice if we could get it working because we would save compute when doing inference.

[nhnt11]

@AmericanPresidentJimmyCarter Noob question - what is "FID"?

[nhnt11]

Never mind - found it! https://en.wikipedia.org/wiki/Fr%C3%A9chet_inception_distance

[AmericanPresidentJimmyCarter]

Yeah, when training new text to image models we always use thing sampler because it produces the best (lowest) FID values. It works very well for SD1.x/2, so it would be good to figure out what is causing the issue with SDXL.

[CodeCorrupt]

Hey @yiyixuxu It would be great if we could find the root cause and get the DPM++ schedulers to "work super great" on SDXL 😄. I'm doing some research myself, but I imagine it would be far more efficient if you and the team could look into it since you have the domain knowledge here.
My biggest question still is, "Why don't DPM++ schedulers work well?".

[LuChengTHU]

DPM++ scheduler is known to not work super well for SDXL. Euler is usually the better choice

Hi @patrickvonplaten , is there any more examples / findings about this conclusion? I will try to figure out the reason :)

[patrickvonplaten]

DPM++ scheduler is known to not work super well for SDXL. Euler is usually the better choice

Hi @patrickvonplaten , is there any more examples / findings about this conclusion? I will try to figure out the reason :)

I'm not fully sure why it happens. The effect seems to go away a bit when using higher inference step sizes (e.g. 50), but with just 25 steps there seem to always be some artifacts. It would be incredible if you could dive a bit deeper here ❤️

[LuChengTHU]

Hi guys, I've found the reason and created a PR to fix this issue. I think now DPM++ can work greatly well for SDXL :)

Please check and try this: #5541