Support convert LoRA safetensors into diffusers format

[haofanwang]

This PR hanldes #2363 and part of #2326.

In summary, given a LoRA weight in safetensors format, we can merge it into a stable diffusion model in diffusers. It is quite easy to use as below. I have tested it with civitai models.

python convert_lora_safetensor_to_diffusers.py --base_model_path runwayml/stable-diffusion-v1-5 --checkpoint_path xxx.safetensors --dump_path ./outputs

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[haofanwang]

I have run quality checker on locally, it should be fine to merge.

[garyhxfang]

@haofanwang Hi, master. If I would like to bake 2 LoRA into model, can I just run the script two times?:
first time bake the LoRA X into model A to generate model B,
and the second time bake the LoRA Y into model B to get model C.

[haofanwang]

@garyhxfang Yes, I did try this using two different LoRA (as style insertion), it would work.

[jndietz]

@haofanwang This is awesome.

This is kind of a loaded question, and not really related to the PR, but how did you "know" what to code for this PR? How can I learn more about the structure of pytorch, safetensors, and diffusers binaries and use them effectively?

[garyhxfang]

@haofanwang Hi, I try the script, and it work well currently. Thanks a lot for your great work!
Only a small mistake
parser.add_argument("--alpha", default=0.75, type=int, help="The merging ratio in W = W0 + alpha * deltaW")
Think the type of alpha should be float? I can run after I change it to float.

[haofanwang]

@garyhxfang I'm glad to know it is helpful to you! I have fixed this typo, thanks.

@patrickvonplaten Could you take a look for this PR so that we merge it ASAP. It seems that many users are waiting for it.

[patrickvonplaten]

I don't fully understand this - why do are we updating the layer weight here?

[patrickvonplaten]

Ah I see! Smart! Ok cool, let's go for it

[leebory]

excuse,me. when i run curr_layer.weight.data += alpha * torch.mm(weight_up, weight_down).unsqueeze(2).unsqueeze(3)` it shows error like "'Modulist' object has no attribute 'weight'"，May I ask if you know how to solve it？

[AlexxxStarkkk]

@haofanwang Hi！
I used the “convert_lora_safetensor_to_diffusers.py” to convert the [one-piece-wano-saga-style-lora]--“wanostyle_2_offset.safesensor” to a folder(size about 5G)。
then, i used it:
`import torch, os
from diffusers import StableDiffusionPipeline, DPMSolverMultistepScheduler
from diffusers.models import AutoencoderKL

os.environ["CUDA_VISIBLE_DEVICES"] = "0"

model_id = r".\sd_model\stable-diffusion-v1-5" # local sd model
vae = AutoencoderKL.from_pretrained("stabilityai/sd-vae-ft-mse", cache_dir=r"D:\sd310\vae_model")

pipe = StableDiffusionPipeline.from_pretrained(model_id)

lora_model_path = r'D:\sd310\lora_model\diffusers_format\one_piece_diffusers_model'
pipe.unet.load_attn_procs(lora_model_path)

pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)

pipe = pipe.to("cuda")

prompt = "1boy, wanostyle, monkey d luffy, smiling"

image = pipe(prompt, height=504, width=680).images[0]
image.save("./001_images_test/one_piece1.png")`

but there is an error:
Traceback (most recent call last): File "D:\sd310\main_test.py", line 69, in <module> pipe.unet.load_attn_procs(lora_model_path) File "E:\programs\Anoconda3\envs\sd310\lib\site-packages\diffusers\loaders.py", line 204, in load_attn_procs model_file = _get_model_file( File "E:\programs\Anoconda3\envs\sd310\lib\site-packages\diffusers\utils\hub_utils.py", line 275, in _get_model_file raise EnvironmentError( OSError: Error no file named pytorch_lora_weights.bin found in directory D:\sd310\lora_model\diffusers_format\one_piece_diffusers_model.

then i use this code:https://github.com/haofanwang/Lora-for-Diffusers/blob/main/format_convert.py
i create a "onepiece.bin", then load it——— "pipe.unet.load_attn_procs(r'D:\sd310\lora_model\diffusers_format\onepiece.bin')"

but still not work

how can i fix it?

[leebory]

excuse,me. when i run curr_layer.weight.data += alpha * torch.mm(weight_up, weight_down).unsqueeze(2).unsqueeze(3)` it shows error like "'Modulist' object has no attribute 'weight'"，May I ask if you know how to solve it？