LoRA from train_dreambooth_lora_sdxl.py is not working in A1111 anymore

[patryk-bartkowiak-nitid]

I have been using train_dreambooth_lora_sdxl.py and convert_diffusers_sdxl_lora_to_webui.py to train LoRA for specific character, It was working till like a week ago. I am using the same baseline model and the same data.

I realized that previous size of all the LoRA files had 29967176 bytes, now it has 29889672 and less keys in dict after I load it as pure .safetensors file.

I realized that it works fine with inference guide in README:

import torch
from diffusers import DiffusionPipeline

pretrained_model = "./pretrained_models/dreamshaper-xl"
lora_weights = "./outputs/dreamshaper-xl_claire/checkpoint-2000/"

prompt = "photo of wff woman, sitting in train"
negative_prompt = "text, watermark, low quality, medium quality, blurry, censored, wrinkles, deformed, mutated text, watermark, low quality, medium quality, blurry, censored, wrinkles, deformed, mutated"

pipe = DiffusionPipeline.from_pretrained(pretrained_model, torch_dtype=torch.float16)
pipe = pipe.to("cuda")
pipe.load_lora_weights(lora_weights)

image = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    num_inference_steps=50,
    seed=420,
).images[0]

image.save("lora_inference.png")

But after I convert and load to A1111 (it loads correctly) it doesnt work anymore, looks like its adding some noise to the output only.

I already tried checkpointing to previous commits on diffusers, torch and torchvision, but nothing really helps. I am still not able to use LoRA in A1111.

Code to train LoRA:

export MODEL_NAME="pretrained_models/dreamshaper-xl"
export INSTANCE_DIR="data/claire"
export MAX_TRAIN_STEPS=5000
export CHECKPOINTING_STEPS=500


export OUTPUT_DIR="outputs/$(basename ${MODEL_NAME})_$(basename ${INSTANCE_DIR})_tmp"
export CUDA_LAUNCH_BLOCKING=1
export TORCH_USE_CUDA_DSA=1

printf "\n\nTraining Claire model with $MODEL_NAME on $INSTANCE_DIR, saving to $OUTPUT_DIR\n\n"

accelerate launch diffusers/examples/dreambooth/train_dreambooth_lora_sdxl.py \
	--instance_prompt="photo of wff woman, isolated on white background" \
	--pretrained_model_name_or_path=$MODEL_NAME \
	--instance_data_dir=$INSTANCE_DIR \
	--output_dir=$OUTPUT_DIR \
	--resolution=1024 \
	--train_batch_size=2 \
	--gradient_accumulation_steps=4 \
	--learning_rate=1e-4 \
	--lr_scheduler="constant" \
	--lr_warmup_steps=0 \
	--max_train_steps=$MAX_TRAIN_STEPS \
	--seed="0" \
	--train_text_encoder \
	--enable_xformers_memory_efficient_attention \
	--gradient_checkpointing \
	--use_8bit_adam \
	--checkpointing_steps=$CHECKPOINTING_STEPS

Code to convert to A1111 format

python /project/diffusers/scripts/convert_diffusers_sdxl_lora_to_webui.py {input_path} {output_path}

System info

- `diffusers` version: 0.26.0.dev0
- Platform: Linux-5.15.0-92-generic-x86_64-with-glibc2.27
- Python version: 3.10.9
- PyTorch version (GPU?): 2.0.0 (True)
- Huggingface_hub version: 0.20.3
- Transformers version: 4.37.2
- Accelerate version: 0.26.1
- xFormers version: 0.0.19
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

[sayakpaul]

No need to duplicate the issues and discussions 😅