v20.8.0

• 2023/02/22 (v20.8.0):
  • Add gui support for optimizers: AdamW, AdamW8bit, Lion, SGDNesterov, SGDNesterov8bit, DAdaptation, AdaFactor
  • Add gui support for --noise_offset
  • Refactor optmizer options. Thanks to mgz-dev!
    • Add --optimizer_type option for each training script. Please see help. Japanese documentation is here.
    • --use_8bit_adam and --use_lion_optimizer options also work and will override the options above for backward compatibility.
  • Add SGDNesterov and its 8bit.
  • Add D-Adaptation optimizer. Thanks to BootsofLagrangian and all!
    • Please install D-Adaptation optimizer with pip install dadaptation (it is not in requirements.txt currently.)
    • Please see kohya-ss/sd-scripts#181 for details.
  • Add AdaFactor optimizer. Thanks to Toshiaki!
  • Extra lr scheduler settings (num_cycles etc.) are working in training scripts other than train_network.py.
  • Add --max_grad_norm option for each training script for gradient clipping. 0.0 disables clipping.
  • Symbolic link can be loaded in each training script. Thanks to TkskKurumi!