interrupt function for gui (feature request)

[Pychnight]

it seems training is incredibly slow, i have been testing different batches and data sets, it's running at 1.29it/s per second compared to atomics1111 gui dream booth training.

so, far I'm enjoying the flexibility of your script but i feel it needs the ability to specify save points at number of steps on top of the epochs and would be nice if there was a interrupt command to tell it to save right at the current step and stop training.

Would be useful.

Generate image preview of all concepts every X steps (allows the user to decide when the model is fully trained and prevent over training)
Allow to interrupt / save at current step though gui / python script
Allow save state to support saving at X number of steps (this could be set the same amount as the generate preview)

[bmaltais]

For some of those recommendations it would require modifications of the code maintained by kohya in his repo. You should ask him to implement them and once he does I will be able to wrap them with a GUI... Here is where he maintain the actual python code: https://github.com/kohya-ss/sd-scripts

[bmaltais]

If anyone know how to implement the interrupt function in python please let me know... I know enough python just to be dangerous ;-) but I would gladly take a pull request to implement the interrupt function.

I might study auto1111 python code to see how he does it.

[mykeehu]

I hope a solution can be found, because currently I can interrupt the training with CTRL+C, but this also stops the whole server. You might also want to look at interrupting D8, because there is also the possibility to save the model in case of an interruption.

[bmaltais]

One day I might research the subject and see if it would be implemented... But since the GUI is just a layer on top of the various CLI I am not sure how that would work. At 1st kohya would need to implement it in his own code. When he does I might be able to make it work by passing down the ctrl-c to his cli...

[ruSauron]

I don't know if that helps or not. There is an extension for automatic1111 that has an interrupt button:

https://github.com/vladmandic/sd-extension-system-info

[ruSauron]

Sorry, I didn't mean it was already done. That extension only works in automatic1111 webui, it doesn't allow to interrupt the process, it only displays a button.

The button uses modules\shared.py, which provides access to state. State uses the modules\processing.py, which is the line

    with torch.no_grad(), p.sd_model.ema_scope():
        With devices.autocast():
            p.init(p.all_prompts, p.all_seeds, p.all_subseeds)
...
            if state.interrupted:
                break

It's not the gradio itself that interrupts the process, but the module in automatic1111 webui, so my comment above is pretty useless