[Bug]: Generating using LoRA fails with Runtime Error with `Lora/Networks: use old method` enabled #12104

JaimeBorondo · 2023-07-27T15:12:06Z

Is there an existing issue for this?

I have searched the existing issues and checked the recent builds/commits

What happened?

I'm on commit 68f336b, master HEAD at time of posting.
None of my LORAs seem to be working anymore. Normal prompting works fine, but as soon as I try generating after adding a LORA to my prompt I receive the following:

RuntimeError: mat1 and mat2 shapes cannot be multiplied (77x3072 and 768x128)

I'm not well versed in python nor the inner workings of stable diffusion, so I can't debug this myself effectively.
I don't think providing the LORA file or prompt is necessary, as I can reproduce this with any combination of checkpoints and LORAs which would previously work fine.

Steps to reproduce the problem

Select any SD checkpoint
txt2image tab
Any combination of prompt and negative prompt, doesnt seem to matter
Add a LORA to the prompt, no need to even add the activation token.
Any generation settings (for my tests I'm using Euler a, 20 steps, 512x512, CFG 7, no scripts, no hires. fix, no face restore).
Generate

What should have happened?

I would expect the LORA to perform as it did in earlier versions with the same configuration, at the very least, generate an image. I haven't done a bisect, but I tried a commit from a week ago or so and it worked fine there. Every time I pull I delete venv and repositories folders beforehand.

Version or Commit where the problem happens

68f336b

What Python version are you running on ?

Python 3.10.x

What platforms do you use to access the UI ?

Windows

What device are you running WebUI on?

Nvidia GPUs (RTX 20 above)

Cross attention optimization

xformers

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

--xformers --reinstall-xformers --precision full --no-half --skip-torch-cuda-test --opt-split-attention

List of extensions

ddetailer, sd-webui-supermerger, stable-diffusion-webui-dataset-tag-editor, stable-diffusion-webui-wd14-tagger

Console logs

venv "C:\sd\sdwebui\webui\venv\Scripts\Python.exe"
Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)]
Version: v1.5.1
Commit hash: 68f336bd994bed5442ad95bad6b6ad5564a5409a
Installing xformers
Collecting xformers==0.0.20
  Using cached xformers-0.0.20-cp310-cp310-win_amd64.whl (97.6 MB)
Installing collected packages: xformers
Successfully installed xformers-0.0.20

[notice] A new release of pip available: 22.3.1 -> 23.2
[notice] To update, run: C:\sd\sdwebui\webui\venv\Scripts\python.exe -m pip install --upgrade pip

Launching Web UI with arguments: --xformers --reinstall-xformers --precision full --no-half --skip-torch-cuda-test --opt-split-attention
Check config files...
Done
Loading weights [cb15a7187a] from C:\sd\sdwebui\webui\models\Stable-diffusion\Deliberate-inpainting.safetensors
Creating model from config: C:\sd\sdwebui\webui\configs\v1-inpainting-inference.yaml
LatentInpaintDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.54 M params.
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 84.3s (launcher: 11.6s, import torch: 31.4s, import gradio: 8.7s, setup paths: 10.6s, other imports: 8.2s, opts onchange: 0.4s, setup codeformer: 0.4s, list SD models: 0.3s, load scripts: 11.1s, create ui: 1.0s, gradio launch: 0.5s).
Applying attention optimization: xformers... done.
Model loaded in 10.0s (load weights from disk: 0.8s, create model: 1.2s, apply weights to model: 6.4s, move model to device: 1.5s).
Loading weights [f36b3ca4d1] from C:\sd\sdwebui\webui\models\Stable-diffusion\edgeOfRealism_edgeOfRealismBakedVAE.safetensors
Creating model from config: C:\sd\sdwebui\webui\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Applying attention optimization: xformers... done.
Model loaded in 8.3s (create model: 0.6s, apply weights to model: 5.9s, move model to device: 1.7s).
*** Error completing request
*** Arguments: ('task(yot3zok0bchp1w0)', 'pov of a beautiful asian woman, formal dress, perfect eyes, petite body, in the forest, colorful, yellow leaves, autumn, hair bun, black hair, facing the viewer, bokeh, soft lighting, perfect face, eye contact, brown eyes, <lora:iu_eor_new-000017:1>', 'badhandv4 easynegative ng_deepnegative_v1_75t', [], 20, 0, False, False, 1, 1, 7, -1.0, -1.0, 0, 0, 0, False, 512, 512, False, 0.7, 2, 'Latent', 0, 0, 0, 0, '', '', [], <gradio.routes.Request object at 0x000001E14BCBFBB0>, 0, False, False, 'positive', 'comma', 0, False, False, '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, '', 'None', 30, 4, 0, 0, False, 'None', '<br>', 'None', 30, 4, 0, 0, 4, 0.4, True, 32) {}
    Traceback (most recent call last):
      File "C:\sd\sdwebui\webui\modules\call_queue.py", line 58, in f
        res = list(func(*args, **kwargs))
      File "C:\sd\sdwebui\webui\modules\call_queue.py", line 37, in f
        res = func(*args, **kwargs)
      File "C:\sd\sdwebui\webui\modules\txt2img.py", line 62, in txt2img
        processed = processing.process_images(p)
      File "C:\sd\sdwebui\webui\modules\processing.py", line 677, in process_images
        res = process_images_inner(p)
      File "C:\sd\sdwebui\webui\modules\processing.py", line 783, in process_images_inner
        p.setup_conds()
      File "C:\sd\sdwebui\webui\modules\processing.py", line 1191, in setup_conds
        super().setup_conds()
      File "C:\sd\sdwebui\webui\modules\processing.py", line 364, in setup_conds
        self.uc = self.get_conds_with_caching(prompt_parser.get_learned_conditioning, negative_prompts, self.steps * self.step_multiplier, [self.cached_uc], self.extra_network_data)
      File "C:\sd\sdwebui\webui\modules\processing.py", line 353, in get_conds_with_caching
        cache[1] = function(shared.sd_model, required_prompts, steps)
      File "C:\sd\sdwebui\webui\modules\prompt_parser.py", line 163, in get_learned_conditioning
        conds = model.get_learned_conditioning(texts)
      File "C:\sd\sdwebui\webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 669, in get_learned_conditioning
        c = self.cond_stage_model(c)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\sdwebui\webui\modules\sd_hijack_clip.py", line 234, in forward
        z = self.process_tokens(tokens, multipliers)
      File "C:\sd\sdwebui\webui\modules\sd_hijack_clip.py", line 271, in process_tokens
        z = self.encode_with_transformers(tokens)
      File "C:\sd\sdwebui\webui\modules\sd_hijack_clip.py", line 324, in encode_with_transformers
        outputs = self.wrapped.transformer(input_ids=tokens, output_hidden_states=-opts.CLIP_stop_at_last_layers)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 811, in forward
        return self.text_model(
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 721, in forward
        encoder_outputs = self.encoder(
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 650, in forward
        layer_outputs = encoder_layer(
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 389, in forward
        hidden_states = self.mlp(hidden_states)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 344, in forward
        hidden_states = self.fc1(hidden_states)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\sdwebui\webui\extensions-builtin\Lora\networks.py", line 357, in network_Linear_forward
        return network_forward(self, input, torch.nn.Linear_forward_before_network)
      File "C:\sd\sdwebui\webui\extensions-builtin\Lora\networks.py", line 345, in network_forward
        y = module.forward(y, input)
      File "C:\sd\sdwebui\webui\extensions-builtin\Lora\network_lora.py", line 84, in forward
        return y + self.up_model(self.down_model(x)) * self.multiplier() * self.calc_scale()
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\sdwebui\webui\extensions-builtin\Lora\networks.py", line 357, in network_Linear_forward
        return network_forward(self, input, torch.nn.Linear_forward_before_network)
      File "C:\sd\sdwebui\webui\extensions-builtin\Lora\networks.py", line 337, in network_forward
        y = original_forward(module, input)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\linear.py", line 114, in forward
        return F.linear(input, self.weight, self.bias)
    RuntimeError: mat1 and mat2 shapes cannot be multiplied (77x3072 and 768x128)

Additional information

None of the extensions listed are used in the context of this issue

The text was updated successfully, but these errors were encountered:

fhnb16 · 2023-07-27T19:47:13Z

Same issue

aunymoons · 2023-07-27T21:52:22Z

Same issue, heres is my infodump, i just upgraded everything, xformers, every plugin, every extra is turned off and i still cannot get regular loras to work, same error:


*** Error completing request
*** Arguments: ('task(hzxoeru4futcnft)', 'test face with detail andtat <lora:5shot:1>', '', [], 20, 0, False, False, 1, 1, 7, -1.0, -1.0, 0, 0, 0, False, 768, 512, False, 0.7, 2, 'Latent', 0, 0, 0, 0, '', '', [], <gradio.routes.Request object at 0x000001D023939B10>, 0, True, False, 1, False, False, False, 1.1, 1.5, 100, 0.7, False, False, True, False, False, 0, 'Gustavosta/MagicPrompt-Stable-Diffusion', '', <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000001D0280573A0>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000001D028056440>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000001D02814BD00>, False, '1:1,1:2,1:2', '0:0,0:0,0:1', '0.2,0.8,0.8', 150, 0.2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, False, False, False, 'positive', 'comma', 0, False, False, '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, None, None, False, None, None, False, None, None, False, 50, '', 2, 4.0, False, '', '', False, False, False, False, 'Illustration', 'svg', True, True, False, 0.5, False, 16, True, 16) {}
    Traceback (most recent call last):
      File "E:\SD\stable-diffusion-webui\modules\call_queue.py", line 58, in f
        res = list(func(*args, **kwargs))
      File "E:\SD\stable-diffusion-webui\modules\call_queue.py", line 37, in f
        res = func(*args, **kwargs)
      File "E:\SD\stable-diffusion-webui\modules\txt2img.py", line 62, in txt2img
        processed = processing.process_images(p)
      File "E:\SD\stable-diffusion-webui\modules\processing.py", line 677, in process_images
        res = process_images_inner(p)
      File "E:\SD\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 42, in processing_process_images_hijack
        return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
      File "E:\SD\stable-diffusion-webui\modules\processing.py", line 783, in process_images_inner
        p.setup_conds()
      File "E:\SD\stable-diffusion-webui\modules\processing.py", line 1191, in setup_conds
        super().setup_conds()
      File "E:\SD\stable-diffusion-webui\modules\processing.py", line 364, in setup_conds
        self.uc = self.get_conds_with_caching(prompt_parser.get_learned_conditioning, negative_prompts, self.steps * self.step_multiplier, [self.cached_uc], self.extra_network_data)
      File "E:\SD\stable-diffusion-webui\modules\processing.py", line 353, in get_conds_with_caching
        cache[1] = function(shared.sd_model, required_prompts, steps)
      File "E:\SD\stable-diffusion-webui\modules\prompt_parser.py", line 163, in get_learned_conditioning
        conds = model.get_learned_conditioning(texts)
      File "E:\SD\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 669, in get_learned_conditioning
        c = self.cond_stage_model(c)
      File "E:\SD\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "E:\SD\stable-diffusion-webui\modules\sd_hijack_clip.py", line 234, in forward
        z = self.process_tokens(tokens, multipliers)
      File "E:\SD\stable-diffusion-webui\modules\sd_hijack_clip.py", line 271, in process_tokens
        z = self.encode_with_transformers(tokens)
      File "E:\SD\stable-diffusion-webui\modules\sd_hijack_clip.py", line 324, in encode_with_transformers
        outputs = self.wrapped.transformer(input_ids=tokens, output_hidden_states=-opts.CLIP_stop_at_last_layers)
      File "E:\SD\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "E:\SD\stable-diffusion-webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 822, in forward
        return self.text_model(
      File "E:\SD\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "E:\SD\stable-diffusion-webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 740, in forward
        encoder_outputs = self.encoder(
      File "E:\SD\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "E:\SD\stable-diffusion-webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 654, in forward
        layer_outputs = encoder_layer(
      File "E:\SD\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "E:\SD\stable-diffusion-webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 393, in forward
        hidden_states = self.mlp(hidden_states)
      File "E:\SD\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "E:\SD\stable-diffusion-webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 348, in forward
        hidden_states = self.fc1(hidden_states)
      File "E:\SD\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "E:\SD\stable-diffusion-webui\extensions-builtin\Lora\networks.py", line 357, in network_Linear_forward
        return network_forward(self, input, torch.nn.Linear_forward_before_network)
      File "E:\SD\stable-diffusion-webui\extensions-builtin\Lora\networks.py", line 345, in network_forward
        y = module.forward(y, input)
      File "E:\SD\stable-diffusion-webui\extensions-builtin\Lora\network_lora.py", line 84, in forward
        return y + self.up_model(self.down_model(x)) * self.multiplier() * self.calc_scale()
      File "E:\SD\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "E:\SD\stable-diffusion-webui\extensions-builtin\Lora\networks.py", line 357, in network_Linear_forward
        return network_forward(self, input, torch.nn.Linear_forward_before_network)
      File "E:\SD\stable-diffusion-webui\extensions-builtin\Lora\networks.py", line 337, in network_forward
        y = original_forward(module, input)
      File "E:\SD\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\linear.py", line 114, in forward
        return F.linear(input, self.weight, self.bias)
    RuntimeError: mat1 and mat2 shapes cannot be multiplied (77x3072 and 768x8)

also my current versions of dependencies:

[+] xformers version 0.0.20 installed.
[+] torch version 2.0.1+cu118 installed.
[+] torchvision version 0.15.2+cu118 installed.
[+] accelerate version 0.19.0 installed.
[+] diffusers version 0.16.1 installed.
[+] transformers version 4.30.2 installed.
[+] bitsandbytes version 0.35.4 installed.

ehowton · 2023-07-28T03:31:30Z

I am experiencing this issue as well.

NefersQ · 2023-07-28T15:13:50Z

same problem

aunymoons · 2023-07-28T15:16:02Z

seems a duplicate of:

#12127

NefersQ · 2023-07-28T15:50:18Z

Restored june version of WebUI and extensions and all is working fine

aunymoons · 2023-07-29T12:52:08Z

Yeah but i cannot be switching branches between that june version that is incompatible with SDXL and the current release, would be nice to have both working.

I think here is another duplicate btw:

#12157

AvaCadava · 2023-08-06T12:37:58Z

same problem (SD1,5 models + LORA)

aunymoons · 2023-08-08T21:33:04Z

Ok so, i managed to solve it, there is a setting that says:

Lora/Networks: use old method that takes longer when you have multiple Loras active and produces same results as kohya-ss/sd-webui-additional-networks extension

in the settings ( settings -> show all -> find "LORA" string for quick getting there )

if this is enabled, it causes this issue and wont allow you to use Loras, just disable it and it should work ( it worked for me )

catboxanon · 2023-08-11T01:25:21Z

~~Could someone who is experiencing this issue link to the LoRA they are using? I can't reproduce this with the Lora/Networks compatbility setting mentioned above.~~ Nevermind, can reproduce it now. Regional Prompter was overriding the setting for me.

@hako-mikan also suggests this option is required for some extensions, per #12157.

When using an extension that requires changing the strength of LoRA at each step, it is necessary to enable this option; otherwise, LoRA and Model reloading will be performed every step, resulting in significantly slower generation times

catboxanon · 2023-08-11T02:23:44Z

Did a bisect, it was introduced by the first commit as part of the extra network rework. b75b004

#11821

It seems like before this commit the shape passed into the forward function is fine (torch.Size([1, 77, 768])) but after this commit it eventually (in my test at the 21st iteration) passes in something much larger (torch.Size([1, 77, 3072]))

catboxanon · 2023-08-11T03:44:50Z

#12466 should close this

JaimeBorondo added the bug-report Report of a bug, yet to be confirmed label Jul 27, 2023

akx added the sdxl Related to SDXL label Jul 31, 2023

catboxanon removed the sdxl Related to SDXL label Aug 7, 2023

JaimeBorondo mentioned this issue Aug 10, 2023

[Bug]: Can't use any lora since last pull #12464

Closed

1 task

catboxanon mentioned this issue Aug 11, 2023

[Bug]: SDXL1.0 RuntimeError: mat1 and mat2 must have the same dtype #12127

Closed

1 task

catboxanon mentioned this issue Aug 11, 2023

[Bug]: Error with Lora/Networks: use old method enabled in Ver. 1.5 or later #12157

Closed

1 task

catboxanon added bug Report of a confirmed bug and removed bug-report Report of a bug, yet to be confirmed labels Aug 11, 2023

catboxanon changed the title ~~[Bug]: Generating using LORA fails with Runtime Error~~ [Bug]: Generating using LoRA fails with Runtime Error with Lora/Networks: use old method enabled Aug 11, 2023

This was referenced Aug 11, 2023

Fix to make Lora/Networks: use old method setting work #12465

Closed

Fix broken Lora/Networks: use old method option #12466

Merged

catboxanon linked a pull request Aug 11, 2023 that will close this issue

Fix broken Lora/Networks: use old method option #12466

Merged

4 tasks

catboxanon closed this as completed Aug 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: Generating using LoRA fails with Runtime Error with `Lora/Networks: use old method` enabled #12104

[Bug]: Generating using LoRA fails with Runtime Error with `Lora/Networks: use old method` enabled #12104

JaimeBorondo commented Jul 27, 2023 •

edited

Loading

fhnb16 commented Jul 27, 2023

aunymoons commented Jul 27, 2023

ehowton commented Jul 28, 2023

NefersQ commented Jul 28, 2023

aunymoons commented Jul 28, 2023

NefersQ commented Jul 28, 2023

aunymoons commented Jul 29, 2023

AvaCadava commented Aug 6, 2023

aunymoons commented Aug 8, 2023

catboxanon commented Aug 11, 2023 •

edited

Loading

catboxanon commented Aug 11, 2023 •

edited

Loading

catboxanon commented Aug 11, 2023

[Bug]: Generating using LoRA fails with Runtime Error with Lora/Networks: use old method enabled #12104

[Bug]: Generating using LoRA fails with Runtime Error with Lora/Networks: use old method enabled #12104

Comments

JaimeBorondo commented Jul 27, 2023 • edited Loading

Is there an existing issue for this?

What happened?

Steps to reproduce the problem

What should have happened?

Version or Commit where the problem happens

What Python version are you running on ?

What platforms do you use to access the UI ?

What device are you running WebUI on?

Cross attention optimization

What browsers do you use to access the UI ?

Command Line Arguments

List of extensions

Console logs

Additional information

fhnb16 commented Jul 27, 2023

aunymoons commented Jul 27, 2023

ehowton commented Jul 28, 2023

NefersQ commented Jul 28, 2023

aunymoons commented Jul 28, 2023

NefersQ commented Jul 28, 2023

aunymoons commented Jul 29, 2023

AvaCadava commented Aug 6, 2023

aunymoons commented Aug 8, 2023

catboxanon commented Aug 11, 2023 • edited Loading

catboxanon commented Aug 11, 2023 • edited Loading

catboxanon commented Aug 11, 2023

[Bug]: Generating using LoRA fails with Runtime Error with `Lora/Networks: use old method` enabled #12104

[Bug]: Generating using LoRA fails with Runtime Error with `Lora/Networks: use old method` enabled #12104

JaimeBorondo commented Jul 27, 2023 •

edited

Loading

catboxanon commented Aug 11, 2023 •

edited

Loading

catboxanon commented Aug 11, 2023 •

edited

Loading