[Feature Request]: step by step diffusion

[Ehplodor]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

Sometimes, i get the feeling that diffusion is going in the right direction, preview images are good for a few steps, then some steps later the preview is completely unrecognizable. That is quite disturbing. I believe the sheduled step is too high, and that it crushes the current latents. Therefore, I would like to run diffusion step by step, recording latents in the meantime, in order to be able to come back to an earlier version and altering the scheduling (lowering its value dynamically) in order to better control the diffusion process.

Proposed workflow

1. Go to img2img
2. activate "1-step diffusion" option
3. click "run 1-step" ... n times
4. if needed, adjust diffusion parameters (CFG, denoising strength, ?scheduler?, prompt, ...)
5. continue until the end or until user decides to stop.

Additional information

Relevant (same subject) discussion "image Generate debugger" proposed by @aifartist : #3396

Issue showing sometimes (last image) the step is not optimal, even undesirable : #3483

Here, sometimes the preview images do not change much and it could be desireable to effectively stop generation earlier : #2472

Further discussion about steps in general : #1113 (comment)

[ClashSAN]

I think this sort of thing should be built on the "process_one" commit by @macrosoft
a9e9799
I do not know any scripts/extensions using it yet.

[Ehplodor]

I think this sort of thing should be built on the "process_one" commit by @macrosoft a9e9799 I do not know any scripts/extensions using it yet.

TY @ClashSAN I knew there had been some work in this direction but couldn't find it when searching for relevant info.

[toriato]

import os

from modules import scripts
from modules.sd_samplers import KDiffusionSampler, sample_to_image

orig_callback_state = KDiffusionSampler.callback_state


def callback_state(self: KDiffusionSampler, d: dict) -> None:
    '''
    callback from k-diffusion sampling methods
    https://github.com/crowsonkb/k-diffusion/blob/60e5042ca0da89c14d1dd59d73883280f8fce991/k_diffusion/sampling.py#L67
    '''
    current_step = d['i']
    latent = d["denoised"]

    image = sample_to_image(latent)
    image.save(os.path.join(scripts.basedir(), f"step_{current_step}.png"))

    return orig_callback_state(self, d)


setattr(KDiffusionSampler, "callback_state", callback_state)


class Script (scripts.Script):
    def title(self):
        return "Step by step"

    def show(self, _):
        return scripts.AlwaysVisible

This code might be helpful to make extension.

[Ehplodor]

TY. I looked at your example and other scripts as well for more info however I just don't understand most of it. Frankly speaking, I would really like to help but I can't with my present knowledge. Else I would already have made a PR for this feature as well a a few others. All I can do for the moment is, just like most of us that are more "user" than "devs", give some brain-time to propose ideas that could eventually be implemented, or maybe not ;-) Hope that helps, too.

[aifartist]

I appreciate the response. Since I posted this I've learned more about how the code works. "gradio" which I believe is the GUI toolset used for the UI isn't that well documented and I find it confusing to go in blind into this code. I'm never done HTML, javascript, CSS stuff on top of using gradio. But I did get it working. There is a specific place in the python code where the Skip and Interrupt flags, set by the buttons, are tested for to do their thing. That part of code is easy enough to read. The hard part was learning enough to add an 'Abort' button to the UI. Once I got that working setting a new global variable in "shared.py" was easy. But now I'd have to learn the "development process" for github stuff if I want to contribute. FYI, 2 days ago I learned how an extension worked and added a Debug button which when pressed spits out the current memory usage for both the system as a whole, the GPU, and the current Python process. I did this because I keep running into problem where I run out of memory, and once this happens, my only option is to restart, which takes minutes. I suspect a leak so I'm adding some stuff to see if I can find out why this is happening. Possibly leak?

…

On Tue, Nov 15, 2022 at 2:39 AM Ehplodor ***@***.***> wrote: TY. I looked at your example and other scripts as well for more info however I just don't understand most of it. Frankly speaking, I would really like to help but I can't with my present knowledge. Else I would already have made a PR for this feature as well a a few others. All I can do for the moment is, just like most of us that are more "user" than "devs", give some brain-time to propose ideas that could eventually be implemented, or maybe not ;-) Hope that helps, too. — Reply to this email directly, view it on GitHub <#4709 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A3YFZABZDOECKLN5SYOOG3DWINR6HANCNFSM6AAAAAAR7ZCOSA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[aifartist]

Thanks. This let me learn about other approaches. The idea of a callback makes sense to me. But once I figured out how the "Skip" and "Interrupt" worked I just added an "Abort" option which stops the process BUT skips the saving of the image. IMO, that is what "Skip" should do.

…

On Mon, Nov 14, 2022 at 12:44 PM Sangha Lee ***@***.***> wrote: import os from modules import scriptsfrom modules.sd_samplers import KDiffusionSampler, sample_to_image orig_callback_state = KDiffusionSampler.callback_state def callback_state(self: KDiffusionSampler, d: dict) -> None: ''' callback from k-diffusion sampling methods https://github.com/crowsonkb/k-diffusion/blob/60e5042ca0da89c14d1dd59d73883280f8fce991/k_diffusion/sampling.py#L67 ''' current_step = d['i'] latent = d["denoised"] image = sample_to_image(latent) image.save(os.path.join(scripts.basedir(), f"step_{current_step}.png")) return orig_callback_state(self, d) setattr(KDiffusionSampler, "callback_state", callback_state) class Script (scripts.Script): def title(self): return "Step by step" def show(self, _): return scripts.AlwaysVisible This code might be helpful to you — Reply to this email directly, view it on GitHub <#4709 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A3YFZAEISI7EURF52WTHZJLWIKQCTANCNFSM6AAAAAAR7ZCOSA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ClashSAN]

#5905 (comment)