Possible improvements to DPM++ 2M Karras sampling

[hallatore]

I'm playing around with the sample_dpmpp_2m function in the k-diffusion repo.

I got a quality increase on my images by doing this trick/bug fix?.

I need help to test out if this is just a false positive that seems to work on my machine, or if it works in general.
Please test it out!

How to try out

1. Open stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py

2. Add the following code to the bottom

@torch.no_grad()
def sample_dpmpp_2m_test(model, x, sigmas, extra_args=None, callback=None, disable=None):
    """DPM-Solver++(2M)."""
    extra_args = {} if extra_args is None else extra_args
    s_in = x.new_ones([x.shape[0]])
    sigma_fn = lambda t: t.neg().exp()
    t_fn = lambda sigma: sigma.log().neg()
    old_denoised = None

    for i in trange(len(sigmas) - 1, disable=disable):
        denoised = model(x, sigmas[i] * s_in, **extra_args)
        if callback is not None:
            callback({'x': x, 'i': i, 'sigma': sigmas[i], 'sigma_hat': sigmas[i], 'denoised': denoised})
        t, t_next = t_fn(sigmas[i]), t_fn(sigmas[i + 1])
        h = t_next - t
        
        t_min = min(sigma_fn(t_next), sigma_fn(t))
        t_max = max(sigma_fn(t_next), sigma_fn(t))

        if old_denoised is None or sigmas[i + 1] == 0:
            x = (t_min / t_max) * x - (-h).expm1() * denoised
        else:
            h_last = t - t_fn(sigmas[i - 1])

            h_min = min(h_last, h)
            h_max = max(h_last, h)
            r = h_max / h_min

            h_d = (h_max + h_min) / 2
            denoised_d = (1 + 1 / (2 * r)) * denoised - (1 / (2 * r)) * old_denoised
            x = (t_min / t_max) * x - (-h_d).expm1() * denoised_d

        old_denoised = denoised
    return x

3. Open stable-diffusion-webui\modules\sd_samplers_kdiffusion.py
4. Add the following code next to DPM++ 2M and DPM++ 2M Karras

samplers_k_diffusion = [
    ...
    ('DPM++ 2M Test', 'sample_dpmpp_2m_test', ['k_dpmpp_2m'], {}),
    ('DPM++ 2M Karras Test', 'sample_dpmpp_2m_test', ['k_dpmpp_2m_ka'], {'scheduler': 'karras'}),

[biship]

Nice change, however the modified version is way to washed out for my liking.
Is there a compromise between the two extremes?

[hallatore]

You can adjust the 0.15 factor to anything down to 0.0. Try 0.10 or 0.12 maybe?

[kjerk]

I wouldn't mess with the generation process to control what is effectively just the simplest contrast change. Seems to be around 20% and you're adjusted. Something like:

# As an automated process (needs a flag)
from PIL import Image, ImageEnhance
# ... image created earlier
contrasted_img = ImageEnhance.Contrast(orig_image).enhance(1.20)
orig_image.save('{image_path_here}/###-original.png')
contrasted_img.save('{image_path_here}/###-cfix.png')

To automate it, or standalone:

# Dumb simple standalone script, change '/path/to/image.png' and run
import pathlib
from PIL import Image, ImageEnhance

if __name__ == '__main__':
    input_path = pathlib.Path('/path/to/image.png')
    output_path = input_path.parent / f'{input_path.stem}_cfix.png'
    image_orig = Image.open(str(input_path))
    contrasted_img = ImageEnhance.Contrast(image_orig).enhance(1.20)
    contrasted_img.save(str(output_path))

[Panchovix]

@kjerk I'm sorry, but where did you edit those lines/code, to change the contrast?

[catboxanon]

Those are standalone examples.

[bbecausereasonss]

Wow massive quality improvement!

[kjerk]

I was trying to get a better look at this myself to evaluate and did a blowup swapping comparison. Maybe not the best naming but B is for Before Change and A is After Change, using the OP's images, and applying 20% contrast to the after image as per #8457 (reply in thread)

B Before, A After (click for big)

[CoperNick8]

Looks like resolution change - I'm not saying that it it different resolution, but it looks like.

[Miraihi]

I've actually had that problem with seemingly low resolution. So that was the sampler's fault, who would've thought. I thought it's a Vae problem.

[danielos123ssa]

May I ask in what file did you apply the contrast fix? and how did you manage the paths

[CoperNick8]

I like result but I don't like 0.15 in the code. It could be some preference. I have idea to produce series of the images from 0.05, 0.10 to for example 0.50. to get better idea what is this about, but I don't have a time recently, so if somebody likes this idea, then I would be very happy to see the results.

[catboxanon]

Might be something worth bringing up in the k-diffusion repo? The old_denoised variable is not currently exposed, so making an extension for this would require a fork of the repo or a re-definition of the function. I believe an additional callback would also need to be created in the webui in the callback_state function, cfg_denoised_callback happens too late and only contains x.

[EfourC]

@hallatore
How did you get the generated before and after images so precisely matched?

When I did some quick trials, there were always more 'macro' changes between the images that make it hard to compare, rather than only fuzziness/contrast differences like in your examples. By macro, I mean things like foliage pattern in the background, slight shift in stance or hair pattern, and other minor features.

Edit: My test was with xformers disabled.

I didn't see as obvious a difference in the sharpness and contrast like in yours, but the macro changes I ended up with made it hard to really tell.

I'm wondering if the model/vae I used, or some other settings affects this.

Can you give specifics of a prompt, model, etc. so we can compare baselines? (I'll try some more when I have time this evening regardless.)

[hallatore]

Take a look at #8457 (comment)

[Centurion-Rome]

I made some quick tests. It seems to have also some impact on other samplers

Improved quality (wow! It feels like boosting resolution of 1.5 models to resolution of 2.1 :)):

• Euler a
• DPM++ SDE
• DPM++ 2M Karras
• DPM++ SDE Karras

Decreased quality(?):

• DPM++ 2M => big contrast loss after increasing contras image has better quality

Has someone else also tested this change with other samplers?

[Panchovix]

Which lines or how did you edit the code to apply these changes to DPM++ SDE and Euler a?

[Centurion-Rome]

not others as proposed by author ;)

[hallatore]

The sampler is used in some permutations. (The Sampling presets in SD are a mix of sampling + other stuff).
I can't remember exactly which ones use this sampler.

PS: Check comment bellow, I might have found something. 🤔

[Adreitz]

@Centurion-Rome Can you post some examples? Only @Panchvzluck has posted 3rd-party examples so far and I'm not seeing extra detail in them.

[hallatore]

I think I might have found a/the bug.
Please try this code and see if things improve.
I left the changed line commented out.

Replace this in stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py

 outdated ...

[hallatore]

@OEvgeny I tried DPM++ 2M but see perfect results here. Do you have info on how to re-generate that image?

[hallatore]

Here are what I'm seeing.

PS: Need to look at it with 100% zoom.

[hallatore]

Updated the post with proper instructions on how to add this as separate v2 samplers so it's easier to test.

[OEvgeny]

@hallatore I will try to follow the updated method to see if I can come up with the same result I posted earlier

[hallatore]

I see I get washed out results when "Always discard next-to-last sigma" is not checked in settings and using DPM++ 2M.

[Centurion-Rome]

We need to get this fix into official repo :)

[myndxero]

Lol, getting anything into the repo lately has been painfully slow. Dunno if the gitpython PR has been merged yet and that's been a critical issue for weeks! Guess I'll go look now that I'm here.

I respect Auto for the work he's done, but time to pass the torch or at least share it, if gonna be so inactive.

[mcmonkey4eva]

The change described is a shortening of the CFG Scale Scheduler option already included in the DynThresh extension - https://github.com/mcmonkeyprojects/sd-dynamic-thresholding

(EDIT: At the time this was written, this was correct - the opening post just tacked in a scheduler. However the opening post has since been edited to use an entirely different technique)

[Centurion-Rome]

Hmm I don't think it causes such a quality/resolution boost as the solution proposed here.
We need to test it also. Has anyone done this and may share the results?

[myndxero]

And I absolutely adore that extension. I'm still making test grids. Ultimately I think I'm going to leave the scale schedulers on constant. I've started to see a correlation between the ups and ancestral samplers and the downs and non-ancestral samplers, or maybe I have that backwards. Anyway, chosen to just zone in on a happy default spot for the thresh percentile and the CFG/CFG mimic relationship that I like.

Good to see you somewhere other than discord!

[hallatore]

Did a test with low steps. The difference is clearer at lower steps. I feel my version matches the 50 steps version better at lower steps.
PS: Click on image and view it 100% zoomed.

[hallatore]

Here is a version that works with DPM++ 2M. At least I seem to get pretty good results with it.

And with "Always discard next-to-last sigma" turned OFF

At 10 steps: https://imgsli.com/MTYxMjc5

 outdated ...

[fractal-fumbler]

can such improvement applied to another samplers? :)

[Panchovix]

+1 to that, I would like to see it also on DPM++ SDE, DPM++ 2S a and maybe Euler a, but maybe I'm asking too much.

Nice work OP!

[hallatore]

This fix/tweak only works for DPM++ 2M/DPM++ 2M Karras since it's related to how it uses the denoised part from the previous step. It's still work in progress as I'm not sure if I fixed it the right way yet.

I can try to explain what I think is a bug, and what I'm trying to do.

The code uses r = h_last / h which (I guess) presumes that h_last is always larger than h. At lower steps this can be false, especially with DPM++ 2M. So I changed it to r = max(h_last, h) / min(h_last, h).

The other thing is that it user r to make denoised_d a value between denoised and old_denoised. This is then used like this: (-h).expm1() * denoised_d
I changed this so it uses h_d which is a value between h and h_last. Kinda the same way denoised_d was derived.

I'm not sure the logic is correct atm since it's mostly been trial and error. So while I think I need an h_d value, I'm not sure I'm calculating it correctly atm.

Hope that made some sense :)

[fractal-fumbler]

thanks!

[hallatore]

Test with a human face

[catboxanon]

I think @mcmonkey4eva is probably correct. This new enhancement washes out the image when using it with img2img, which the vanilla sampler does not do.

[hallatore]

Did you use the latest code at the initial post? Mine is spot on.

[catboxanon]

You're right, I didn't know the code had been updated since then. Thanks for the pointer!

[FrisanBR]

I didn't find the mentioned paths, has something changed? I started using Stable Diffusion today, so I installed the current version.
"Open stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py"
"Open stable-diffusion-webui\modules\sd_samplers_kdiffusion.py"
I found the sampling.py file in a different path and was able to make the change, but I couldn't find the sd_samplers_kdiffusion.py file

Can anyone help me?

[axsddlr]

in honesty, if you cant find the paths, i wouldnt suggest trying this out until its considered for merge

[OEvgeny]

Is it the same repo? I thought EasyDiffusion is a completely different app not related to stable-diffusion-webui. And these instructions are for the ui available in the repo.

[2blackbar]

This should be in webui by default, so i dont have to put it back in everytime i update

[2blackbar]

i noticed that most devs when they release the code after publishin papers they just move on to next stuff and dont really want to update the code ,probably working on something better, last response on any of the issues with k was in december 2022

[hallatore]

Any way to get this into the code as an alternative version @AUTOMATIC1111 ?
Seems like the k-diffusion repo isn't updated.

[oakif]

I'm wondering this as well. Are there any plans on adding this to A1111 as a v2 or a mod of the original samplers?

[wywywywy]

We need to either fork the k-diffusion repo and become new maintainers, or we take the main bits from the k-diffusion repo and maintain them inside of this repo.

But we may break MacOS compatibility as they use a different k-diffusion.

EDIT: On the other hand, if we take control, we could easily incorporate the MacOS fixes into the new code, which will make it even cleaner.

[wywywywy]

Ok I've gone and forked the k-diffusion repo, and incorporated @hallatore's change plus the MacOS MPS workaround from @brkirch.

To use it, simply delete the repositories/k-diffusion directory, and set these variables in the batch/shell file:

webui-user.bat

set K_DIFFUSION_REPO=https://github.com/wywywywy/k-diffusion.git
set K_DIFFUSION_COMMIT_HASH=ca06f522e6d3f202c25c3565c53afbd9c40ac53d

webui-user.sh

export K_DIFFUSION_REPO="https://github.com/wywywywy/k-diffusion.git"
export K_DIFFUSION_COMMIT_HASH="e3f853a8c9f70052aa1c4bb8cd0e4ec3af7ffaff"

Please can someone with a Mac give a good test.

[nemilya]

If I replace whole repo by your repository (wywywywy/k-diffusion, ca06f5)

• Euler a not working
• PDM++ 2M working (better at small steps)

if use current (brkirch/k-diffusion, 51c977), and overwrite just sampling.py

• Euler a working
• PDM++ 2M working (better at small steps)

[wywywywy]

Any ideas what it could be? The sample_euler_ancestral function in the 2 repos are the same.

https://github.com/brkirch/k-diffusion/blob/c41588ae74e1b51e9f8663106d9843389bf02729/k_diffusion/sampling.py#L139

https://github.com/wywywywy/k-diffusion/blob/ca06f522e6d3f202c25c3565c53afbd9c40ac53d/k_diffusion/sampling.py#L139

[wywywywy]

Actually I just realised that I've missed out @Birch-san's MPS fixes. I'll get them added in and see if they work in this repo.

[wywywywy]

Ok new commit e3f853a8c9f70052aa1c4bb8cd0e4ec3af7ffaff seems to work. Please give it another try.

I was able to use both Euler A and DPM++ 2M Karras on my MBP.

[nemilya]

Just tested new commit - all works great now! Euler A and DPM++ 2M
Thanks!

[generatorman]

What does this change do at a conceptual level?

[wywywywy]

What does this change do at a conceptual level?

Basically @hallatore identified a potential bug in the DPM++ 2M logic, as explained here #8457 (reply in thread)

And from testing it seems to improve the quality quite significantly at low steps.

[hallatore]

The change does two things primarily.

• It makes sure r is always min / max. Since with some steps the next sigma can be higher th
  an the last even though they are normally always lower. This is what causes the artifacts seen especially in DPM++ 2M at low steps.
• The second thing it does it changing h to h_d in the last part. (This part ... (-h_d).expm1() * denoised_d). The reason for this is that denoised_d is calculated based on previous and current step, while h was just "current step". It made sense that both parts are calculated the same way, not only denoised_d.

[nemilya]

The fix works!

Machine: MacbookPro M2 (8Gb, MacOS, Monterey 12.6.3)
Sampler: DPM++ 2M, Steps: 10

Before/after:

I change file sampling.py directly in repository (variant by setting ENV did not change the repository k-diffusion)

ps: another confirmation, same 10 steps (before/after fix)

(examples based on Deliberate model)

[wywywywy]

Cool results thank you. However, colours seem to be less vibrant. Maybe more work is needed.

[thanayut1750]

can we add more contrast to it now? kind of like the result but like you said the colors seem the fade.

[bryanray]

it is not less vibrant, but LESS DETAILS, more blur..... it is not favorable unless you ask for it.

[myndxero]

If ever considered for merge. Yes I'm salty auto mia. Checking out a new fork that's supposed to be pretty active, seems to be getting a lot of attention. https://www.reddit.com/r/StableDiffusion/comments/12grgwh/automatic1111_getting_rusty_future_of_this_repo_i/

…

On Thu, Mar 16, 2023, 03:28 Andre Saddler ***@***.***> wrote: in honesty, if you cant find the paths, i wouldnt suggest trying this out until its considered for merge — Reply to this email directly, view it on GitHub <#8457 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A2755JZHZOK34ASIXX7IS23W4LFLDANCNFSM6AAAAAAVVGRDVQ> . You are receiving this because you commented.Message ID: <AUTOMATIC1111/stable-diffusion-webui/repo-discussions/8457/comments/5330846 @github.com>

[LiteSoul]

I've been using this improvement for over a month... it's great, we need this officially somehow!

[recoilme]

@Metachs: "It looks to me like all you are seeing is faster convergence due to loss of detail" : crowsonkb/k-diffusion#56 (comment)
Take a look at this

[ride5k]

just stumbled across this, and am very happy to have found it!

[bryanray]

the result mostly blurring out lots details.... please fix it ? or is it unfixable....

[ShivaeAI]

I think there are similarities between this new sampling (DPM++ 2M Karras) and kohya_ss's Network Alpha training parameters in terms of adjusting the noise into blur and drastically reducing the noise, but using the flower photos sent from nemilya's comment above, I feel there is a blurred out beauty, yes, only a blurred beauty, no sharpening involved

[Alfonsusnetflix]

i seen that the script is updated ? any possible update ?

[Stalinium-Network]

where do you get the code for DPM 2M Karass, I'm new to this and don't understand how to find the code

[dtlnor]

you may insert the code here modules\sd_samplers_kdiffusion.py
or directly modify from kdiffusion (repositories\k-diffusion\k_diffusion\sampling.py)

note that the implementation here may lost details (crowsonkb/k-diffusion#56 (comment))

[simonmcnair]

Did any submit a PR for this ? Seems quite trivial to do ?