Diffusers 0.24.0 causing clip_skip error when generating a sample during training

[sangoi-exe]

The 0.24.0 version of Diffusers used in the dev2 branch is causing an error in clip_skip when generating a sample.

I changed the requirements.txt to the version from the master branch (0.21.4) and the problem was solved.

I even saw an issue from 2 weeks ago reporting this error, but it mistakenly claimed it was an error when saving the checkpoint. However, the checkpoint and the sample were being generated in the same step, the error is indeed when generating the sample.

[bmaltais]

This is kind of anoying as the current diffusers module version is not compatible with the convert model to LCM model that I added and I don't like the idea of wasting all this time for nothing... I wonder if kohya could not fix the issue on his side... what is the error when you try to generate samples? Maybe it is an easy fix... this is frustrating

[sangoi-exe]

This is kind of anoying as the current diffusers module version is not compatible with the convert model to LCM model that I added and I don't like the idea of wasting all this time for nothing... I wonder if kohya could not fix the issue on his side... what is the error when you try to generate samples? Maybe it is an easy fix... this is frustrating

I even tried to solve it by looking up what had changed between versions of the function that causes the error, but it didn't work out very well.

Below is the error:

generating sample images at step / サンプル画像生成 ステップ: 10
C:\kohya_ss\venv\lib\site-packages\diffusers\pipelines\stable_diffusion\pipeline_stable_diffusion.py:181: FutureWarning: The configuration file of this scheduler: EulerAncestralDiscreteScheduler {
  "_class_name": "EulerAncestralDiscreteScheduler",
  "_diffusers_version": "0.24.0",
  "beta_end": 0.012,
  "beta_schedule": "scaled_linear",
  "beta_start": 0.00085,
  "num_train_timesteps": 1000,
  "prediction_type": "epsilon",
  "steps_offset": 0,
  "timestep_spacing": "linspace",
  "trained_betas": null
}
 is outdated. `steps_offset` should be set to 1 instead of 0. Please make sure to update the config accordingly as leaving `steps_offset` might led to incorrect results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for the `scheduler/scheduler_config.json` file
  deprecate("steps_offset!=1", "1.0.0", deprecation_message, standard_warn=False)
Traceback (most recent call last):
  File "C:\kohya_ss\train_network.py", line 1012, in <module>
    trainer.train(args)
  File "C:\kohya_ss\train_network.py", line 847, in train
    self.sample_images(accelerator, args, None, global_step, accelerator.device, vae, tokenizer, text_encoder, unet)
  File "C:\kohya_ss\train_network.py", line 131, in sample_images
    train_util.sample_images(accelerator, args, epoch, global_step, device, vae, tokenizer, text_encoder, unet)
  File "C:\kohya_ss\library\train_util.py", line 4452, in sample_images
    return sample_images_common(StableDiffusionLongPromptWeightingPipeline, *args, **kwargs)
  File "C:\kohya_ss\library\train_util.py", line 4551, in sample_images_common
    pipeline = pipe_class(
  File "C:\kohya_ss\library\lpw_stable_diffusion.py", line 535, in __init__
    self.clip_skip = clip_skip
  File "C:\kohya_ss\venv\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 607, in __setattr__
    super().__setattr__(name, value)
AttributeError: can't set attribute 'clip_skip'
steps:   0%|                                                    | 10/41040 [00:44<50:40:53,  4.45s/it, avr_loss=0.0658]
Traceback (most recent call last):
  File "C:\Users\lucas\AppData\Local\Programs\Python\Python3109\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\lucas\AppData\Local\Programs\Python\Python3109\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\kohya_ss\venv\Scripts\accelerate.exe\__main__.py", line 7, in <module>
  File "C:\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 47, in main
    args.func(args)
  File "C:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 986, in launch_command
    simple_launcher(args)
  File "C:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 628, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['C:\\kohya_ss\\venv\\Scripts\\python.exe', './train_network.py', '--enable_bucket', '--min_bucket_reso=256', '--max_bucket_reso=2048', '--pretrained_model_name_or_path=', '--train_data_dir=', '--reg_data_dir=', '--resolution=768,768', '--output_dir=D:/# Stable Diffusion/Kohya/Output', '--logging_dir=D:/# Stable Diffusion/Kohya/Logs', '--network_alpha=1', '--training_comment=', '--save_model_as=safetensors', '--network_module=lycoris.kohya', '--network_args', 'preset=full', 'conv_dim=16', 'conv_alpha=1', 'rank_dropout=0.1', 'module_dropout=0.1', 'use_tucker=False', 'use_scalar=False', 'rank_dropout_scale=True', 'algo=locon', 'train_norm=True', '--text_encoder_lr=1.0', '--unet_lr=1.0', '--network_dim=32', '--output_name=', '--lr_scheduler_num_cycles=40', '--network_dropout=0.1', '--no_half_vae', '--learning_rate=1.0', '--lr_scheduler=cosine', '--train_batch_size=1', '--max_train_steps=41040', '--save_every_n_epochs=1', '--mixed_precision=bf16', '--save_precision=bf16', '--caption_extension=.txt', '--cache_latents', '--cache_latents_to_disk', '--optimizer_type=Prodigy', '--max_grad_norm=1', '--max_data_loader_n_workers=8', '--keep_tokens=1', '--bucket_reso_steps=64', '--flip_aug', '--shuffle_caption', '--xformers', '--persistent_data_loader_workers', '--noise_offset=0.0', '--sample_sampler=euler_a', '--sample_prompts=D:/# Stable Diffusion/Kohya/Output\\sample\\prompt.txt', '--sample_every_n_steps=10']' returned non-zero exit status 1.

[rucola-pizza]

I've encountered an same issue with the StableDiffusionLongPromptWeightingPipeline class in the diffusers library version 0.24. The problem arises when trying to set the clip_skip attribute in the init method of StableDiffusionLongPromptWeightingPipeline.

It seems that the clip_skip attribute cannot be set because there's an existing @Property with the same name in the StableDiffusionPipeline class, which is a superclass of StableDiffusionLongPromptWeightingPipeline. This existing property is read-only as it lacks a setter method.

class StableDiffusionLongPromptWeightingPipeline(StableDiffusionPipeline):
    def __init__(self, ...):
        super().__init__(...)
        self.clip_skip = clip_skip
        ...

class StableDiffusionPipeline(...):
    ...
    @property
    def clip_skip(self):
        return self._clip_skip

[bmaltais]

So... there is no simple fix I guess... I could have implemented a hack to kohya's code until he fix it upstream but given this information it look like this will take a bit more work to solve... If anyone know of to work around this in the current code I can implement the fix in the copy of kohya's code I grab... but until this is possible I will need to think of possible alternatives...

Option 1: Deploy two venv... one following kohya's requirements versions and another with updated diffusers release required for the LCM creation tool... This 2nd could be used for other tools in the future that might require different modules... the downside is this this will lead to another 6GB or so of disk to be consumed and double the setup time... and add complexity to the setup process.

Option 2: Unknown...

[sangoi-exe]

@bmaltais,

Done, I developed a workaround by creating a setter in the StableDiffusionLongPromptWeightingPipeline subclass of lpw_stable_diffusion.py, based on a helpful suggestion from @rucola-pizza.

Additionally, I discovered and added a missing parameter, image_encoder. After solving the clip_skip issue, this problem of the missing parameter emerged. 😅

I have submitted a PR with these changes. The solution seems to effectively address the problem, but it requires further testing.

Modifying the original Kohya file in the dev branch should not present any problems, correct?

[bmaltais]

It will require meticulous merge when Joshua update his code base… and I merge his updates in… so anything you can make to isolate as much as possible from changing his code the best it is.

[bmaltais]

I have merged the PR into dev2, give it a try... if all is good then this could be the solution until kohya fixes his code at the source to work under diffuser 0.24... Thank you @DevArqSangoi for the fix!

[sangoi-exe]

I have merged the PR into dev2, give it a try... if all is good then this could be the solution until kohya fixes his code at the source to work under diffuser 0.24... Thank you @DevArqSangoi for the fix!

@bmaltais,
Great! We can use this workaround until Kohya does the official fix 😁

Np, it was a pleasure to be helpful!