convert_lora_safetensor_to_diffusers

[NicholasKao1029]

this script https://github.com/huggingface/diffusers/blob/main/scripts/convert_lora_safetensor_to_diffusers.py integrates the lora into the Pipeline and then outputs said Pipeline.

Is there a method to turn the .safetensors Lora to a .bin file to be loaded in dynamically similar to the snippet below

from diffusers import StableDiffusionPipeline
import torch

model_path = "sayakpaul/sd-model-finetuned-lora-t4"
pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", torch_dtype=torch.float16)
pipe.unet.load_attn_procs(model_path)
pipe.to("cuda")

prompt = "A pokemon with blue eyes."
image = pipe(prompt, num_inference_steps=30, guidance_scale=7.5, cross_attention_kwargs={"scale": 0.5}).images[0]
image.save("pokemon.png")

As seen here https://huggingface.co/docs/diffusers/training/lora

My complaint is there is a lot of redundancy in the model if you want different loras to be used.

[sayakpaul]

Cc: @patrickvonplaten @Narsil

[Narsil]

The conversion of the LoRA should only be a key remapping right ?

load_attn_procs already supports loading LoRA from safetensors file. I think the problem is that SD-safetensors != diffusers-safetensors. (The name of the tensors are different). Am I correct ?

[NicholasKao1029]

I'm not sure this is the case, as the call to the convert function outputs a Pipe,

diffusers/scripts/convert_lora_safetensor_to_diffusers.py

Line 125 in 1d7b4b6

pipe = convert(base_model_path, checkpoint_path, lora_prefix_unet, lora_prefix_text_encoder, alpha)

This outputs a directory much like https://huggingface.co/runwayml/stable-diffusion-v1-5/tree/main which is a full model

whereas in the load_attn_procs example the use of https://huggingface.co/sayakpaul/sd-model-finetuned-lora-t4/tree/main has a different structure specifically it has pytorch_lora_weights.bin

[NicholasKao1029]

Sorry I misunderstood @Narsil, you are right it is only a key remapping,
However my point still holds true as the output is ultimately a full model

[sayakpaul]

IIUC we output the full model for generality across different pipelines and also to keep the output artifacts to be as self-contained as possible. If you have some ideas on how we can improve this bit, feel free to open a PR :)

@patrickvonplaten Cc.

[fungiboletus]

I was expecting the script to generate a lora and not a whole model. Is it possible to convert the safetensors lora to diffusers lora? Or should we wait for the next release that should support safetensors lora without conversion ?

Edit: I found a conversion script there: https://github.com/haofanwang/Lora-for-Diffusers/blob/18adfa4da0afec46679eb567d5a3690fd6a4ce9c/format_convert.py#L154-L161

[YulienPohl]

Edit: I found a conversion script there: https://github.com/haofanwang/Lora-for-Diffusers/blob/18adfa4da0afec46679eb567d5a3690fd6a4ce9c/format_convert.py#L154-L161

For me this script does not work, it returns a 48 MB bin from a 144 MB safetensor.
And when I use it, it only creates images with colored spots.

[patrickvonplaten]

We should try to have an improved load_lora function that also allows to load from the orig LoRA format (similar to how we do it here: #2009 (comment)).

[NicholasKao1029]

@patrickvonplaten this would be fantastic!

[patrickvonplaten]

Related PR: #2882

[NicholasKao1029]

Updated PR: #2918

[sayakpaul]

Regarding the safetensors LoRA thing, does it make sense?
#2866 (comment)

[NicholasKao1029]

Regarding the safetensors LoRA thing, does it make sense? #2866 (comment)

Sorry for the late reply @sayakpaul.

If I understand the linked comment, that would be for .bin -> safetensors for loras. Is there a corresponding safetensors -> .bin planned here or perhaps I misunderstood

[sayakpaul]

The workflow still remains the same.

You first download the .safetensors file. Let's call the downloaded model path - model_filepath.

Then you load it:

import safetensors

pt_state_dict = safetensors.torch.load_file(model_filepath, device="cpu")

Referencing:

diffusers/src/diffusers/loaders.py

Line 188 in b811964

state_dict = safetensors.torch.load_file(model_file, device="cpu")

Then we can do:

torch.save(pt_state_dict, "pt_state_dict.bin")

And then it should be just:

pipeline.unet.load_attn_procs("pt_state_dict.bin")

A couple of other tings that might be worth noting regarding loading safetensors and operating with it seamlessly.

Note it's also possible to directly load a .safetensors file with load_attn_procs() from the Hugging Face Hub. Let's the HF Hub repo id is: username/repo_containing_lora_files_in_safetensors.

We can then do:

pipeline.unet.load_attn_procs("username/repo_containing_lora_files_in_safetensors", use_safetensors=True)

Explicitly specifying use_safetensors is not necessary because we always prioritize on safetensors by default.

For more details, I welcome you to check out:

diffusers/src/diffusers/loaders.py

Lines 158 to 193 in b811964

	allow_pickle = False
	if use_safetensors is None:
	use_safetensors = is_safetensors_available()
	allow_pickle = True
	user_agent = {
	"file_type": "attn_procs_weights",
	"framework": "pytorch",
	}
	model_file = None
	if not isinstance(pretrained_model_name_or_path_or_dict, dict):
	# Let's first try to load .safetensors weights
	if (use_safetensors and weight_name is None) or (
	weight_name is not None and weight_name.endswith(".safetensors")
	):
	try:
	model_file = _get_model_file(
	pretrained_model_name_or_path_or_dict,
	weights_name=weight_name or LORA_WEIGHT_NAME_SAFE,
	cache_dir=cache_dir,
	force_download=force_download,
	resume_download=resume_download,
	proxies=proxies,
	local_files_only=local_files_only,
	use_auth_token=use_auth_token,
	revision=revision,
	subfolder=subfolder,
	user_agent=user_agent,
	)
	state_dict = safetensors.torch.load_file(model_file, device="cpu")
	except IOError as e:
	if not allow_pickle:
	raise e
	# try loading non-safetensors weights
	pass

I hope this helps.

[NicholasKao1029]

Thanks @sayakpaul for the detailed explanation, I'm running into this issue #3064 I'll follow that one for updates

[sayakpaul]

Closing this issue then. Feel free to re-open.

[ShoufaChen]

Thanks for @sayakpaul's answer. I found that for recent safetensors I should load by:

from safetensors.torch import load_file
model = load_file(xxx)

because directly use import safetensors safetensors.torch.load_file(xxx)
would raise AttributeError: module 'safetensors' has no attribute 'torch'.

Just post here in case it might be helpful.