Flux.1 LoRA training

[bmaltais]

Kohya has added preliminary support for Flux.1 LoRA to his SD3 branch. I have created a sd3-flux.1 branch and updated to the latest sd-scripts sd3 branch code... No GUI integration yet... I will start adding the basic code to be able to establish that the model is Flux as part of the GUI.

[bmaltais]

The branch now contain MVP but for some reason the flux1 trainer crash with an Optimizer argument list is empty.

[bmaltais]

But I am not sure why I keep getting this error when trying to train:

FLUX: Gradient checkpointing enabled.
prepare optimizer, data loader etc.
                    INFO     use 8-bit AdamW optimizer | {}                                                                                                                                  train_util.py:4342
override steps. steps for 4 epochs is / 指定エポックまでのステップ数: 320
enable fp8 training.
Traceback (most recent call last):
  File "D:\kohya_ss\sd-scripts\flux_train_network.py", line 395, in <module>
    trainer.train(args)
  File "D:\kohya_ss\sd-scripts\train_network.py", line 543, in train
    if hasattr(t_enc.text_model, "embeddings"):
  File "D:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1695, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'T5EncoderModel' object has no attribute 'text_model'

Maybe I really need to upgrade to PyTorch to 2.4.0... not liking that as this might bork my non Flux.1 GUI... not feeling like upgrading...

[BenDes21]

But I am not sure why I keep getting this error when trying to train:

FLUX: Gradient checkpointing enabled.
prepare optimizer, data loader etc.
                    INFO     use 8-bit AdamW optimizer | {}                                                                                                                                  train_util.py:4342
override steps. steps for 4 epochs is / 指定エポックまでのステップ数: 320
enable fp8 training.
Traceback (most recent call last):
  File "D:\kohya_ss\sd-scripts\flux_train_network.py", line 395, in <module>
    trainer.train(args)
  File "D:\kohya_ss\sd-scripts\train_network.py", line 543, in train
    if hasattr(t_enc.text_model, "embeddings"):
  File "D:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1695, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'T5EncoderModel' object has no attribute 'text_model'

Maybe I really need to upgrade to PyTorch to 2.4.0... not liking that as this might bork my non Flux.1 GUI... not feeling like upgrading...

Hi there, is it possible to only update PyTorch to 2.4.0 for only the Flux version of Kohya_ss GUI ?

[protector131090]

SimpleTuner upadted to v0.9.8: quantised flux training in 40 gig.. 24 gig.. 16 gig... 13.9 gig.. Waiting so much for kohya

[BenDes21]

SimpleTuner upadted to v0.9.8: quantised flux training in 40 gig.. 24 gig.. 16 gig... 13.9 gig.. Waiting so much for kohya

probably very soon

[BenDes21]

But I am not sure why I keep getting this error when trying to train:

FLUX: Gradient checkpointing enabled.
prepare optimizer, data loader etc.
                    INFO     use 8-bit AdamW optimizer | {}                                                                                                                                  train_util.py:4342
override steps. steps for 4 epochs is / 指定エポックまでのステップ数: 320
enable fp8 training.
Traceback (most recent call last):
  File "D:\kohya_ss\sd-scripts\flux_train_network.py", line 395, in <module>
    trainer.train(args)
  File "D:\kohya_ss\sd-scripts\train_network.py", line 543, in train
    if hasattr(t_enc.text_model, "embeddings"):
  File "D:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1695, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'T5EncoderModel' object has no attribute 'text_model'

Maybe I really need to upgrade to PyTorch to 2.4.0... not liking that as this might bork my non Flux.1 GUI... not feeling like upgrading...

Hi there! Any news about the integration of Flux into the gui ?

[bmaltais]

I am running into similar but different errors. Waiting for the as-scripts code to stabilize to further work on it. I have a lot of the elements already in the gui. The missing ones can be added as extra parameters in the Advanced Accordion.

[BenDes21]

I am running into similar but different errors. Waiting for the as-scripts code to stabilize to further work on it. I have a lot of the elements already in the gui. The missing ones can be added as extra parameters in the Advanced Accordion.

Nice! Should be released soon so, cannot wait to try! Thanks for your work

[bmaltais]

I updated to the latest sd-script commit for flux... still can't run training at my end unfortunately:

FLUX: Gradient checkpointing enabled.
prepare optimizer, data loader etc.
                    INFO     use 8-bit AdamW optimizer | {}                                                                                                                                  train_util.py:4346
override steps. steps for 4 epochs is / 指定エポックまでのステップ数: 320
enable fp8 training.
Traceback (most recent call last):
  File "D:\kohya_ss\sd-scripts\flux_train_network.py", line 397, in <module>
    trainer.train(args)
  File "D:\kohya_ss\sd-scripts\train_network.py", line 543, in train
    if hasattr(t_enc.text_model, "embeddings"):
  File "D:\kohya_ss\venv\lib\site-packages\torch\nn\modules\module.py", line 1695, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'T5EncoderModel' object has no attribute 'text_model'

Here is a copy of my flux1_test.json config if you are interested to poke at it.

flux1_test.json

[bmaltais]

I pushed an update with support for the missing GUI parameters for Flux.1.

Here is the latest config for testing based on Kohya's readme config:

flux1_test.json

[BenDes21]

I pushed an update with support for the missing GUI parameters for Flux.1.

Here is the latest config for testing based on Kohya's readme config:

flux1_test.json

thanks a lot

[bmaltais]

The GUI is a real mess with so many options. I get lost myself when trying to fing where is the option I need to set. Wish there was an easy solution… but I can’t think of one.

[WarAnakin]

The GUI is a real mess with so many options. I get lost myself when trying to fing where is the option I need to set. Wish there was an easy solution… but I can’t think of one.

bro, flux does not support training the text encoder, yet. Set your text encoder lr to 0
That should get you past the error.

[WarAnakin]

Here's some images from a lora i trained

[TripleHeadedMonkey]

I'm surprised that wasn't more widely known actually. Stability AI, when they released SD3, mentioned that training the T5 model was not only not necessary but not recommended.

The same is likely true for FLux also. It simply relies on the tokenization from the Clip L and the Transformer model working in conjunction with the T5 model's established natural language processing.

And Clip L is almost entirely tag-based and seems highly unstable when trained anyway.

In other words as long as you create the embedding within the model itself, the T5's existing capabilities should be enough to hit the ground running and incorporate that embedding into natural language prompting off the bat.

How exactly this translates to the end result is something I am yet to see myself though.

[protector131090]

ai toolkit skript works really great. Trained 3 LORAs so far (in 3 hours) its not perfect but super good. Awaiting for Kohya

[bmaltais]

Here's some images from a lora i trained

Was it trained using kohya_ss? Great results.

[jpXerxes]

Is anyone getting past the AttributeError: 'T5EncoderModel' object has no attribute 'text_model'?
Using the second version of flux1_test, which has lr set to 0 and that doesn't do it.

[b-7777777]

Is anyone getting past the AttributeError: 'T5EncoderModel' object has no attribute 'text_model'? Using the second version of flux1_test, which has lr set to 0 and that doesn't do it.

Someone implemented a potential fix for it, but Kohya hasn't added it yet:
kohya-ss/sd-scripts#1453

[FrakerKill]

The GUI is a real mess with so many options. I get lost myself when trying to fing where is the option I need to set. Wish there was an easy solution… but I can’t think of one.

bro, flux does not support training the text encoder, yet. Set your text encoder lr to 0 That should get you past the error.

Have you an example json?

[jpXerxes]

Someone implemented a potential fix for it, but Kohya hasn't added it yet: kohya-ss/sd-scripts#1453

Koyha has added it now. Pulled these files:
library/flux_train_utils.py
flux_train_network.py
train_network.py
library/flux_models.py

but now I get:
File "E:\kohya_ss\sd-scripts\flux_train_network.py", line 207, in sample_images
accelerator, args, epoch, global_step, flux, ae, text_encoder, self.sample_prompts_te_outputs
AttributeError: 'FluxNetworkTrainer' object has no attribute 'sample_prompts_te_outputs'

[stepfunction83]

I can confirm that I am getting the same AttributeError as @jpXerxes after cloning the latest sd3 branch

Able to bypass the issue and begin training by adding --cache_text_encoder_outputs to the additional parameters!

[jpXerxes]

Able to bypass the issue and begin training by adding --cache_text_encoder_outputs to the additional parameters!

That did it. I ran the test file, with sample outputs every 1 epoch (4 epochs) and prompt: a painting of a steam punk skull with a gas mask , by darius kawasaki

These are the 4 sample images:

[velmbi]

How are you running the sd3_train,py script with kohya? I downloaded it but don't know what to do with it. I've always just used kohya normally but really want to try some flux training.

[jpXerxes]

How are you running the sd3_train,py script with kohya? I downloaded it but don't know what to do with it. I've always just used kohya normally but really want to try some flux training.
Likely you could wait a very short time until bmaltais catches up, but:
I'm no expert, but here's what I did:
First, make sure you have the proper branch of bmaltais/kohya using git checkout sd3-flux.1

Go to https://github.com/kohya-ss/sd-scripts/tree/sd3 and download the 4 files below, and place them in the appropriate folders:
library/flux_train_utils.py
flux_train_network.py
train_network.py
library/flux_models.py

grab the second flux1_test.json posted above in this thread. Edit it to change all hard-coded paths to your own structure. Near the top is the line "additional_parameters": and add to it --cache_text_encoder_outputs

In the gui, add a choice for sample output frequency, and add a prompt

Please, anybody spot something wrong with this please correct me!

[jpXerxes]

A general problem with my test run is that Flux already knows how to deal with that prompt, so I get good images without the Lora. I will have to find something Flux knows nothing about to properly test.

[stepfunction83]

You can also remove the sd scripts directory and replace it with the latest version of the sd3 branch.

…

On Thu, Aug 15, 2024, 3:09 PM jpXerxes ***@***.***> wrote: How are you running the sd3_train,py script with kohya? I downloaded it but don't know what to do with it. I've always just used kohya normally but really want to try some flux training. Likely you could wait a very short time until bmaltais catches up, but: I'm no expert, but here's what I did: First, make sure you have the proper branch of bmaltais/kohya using git checkout sd3-flux.1 Go to https://github.com/kohya-ss/sd-scripts/tree/sd3 and download the 4 files below, and place them in the appropriate folders: library/flux_train_utils.py flux_train_network.py train_network.py library/flux_models.py grab the second flux1_test.json posted above in this thread. Edit it to change all hard-coded paths to your own structure. Near the top is the line "additional_parameters": and add to it --cache_text_encoder_outputs In the gui, add a choice for sample output frequency, and add a prompt Please, anybody spot something wrong with this please correct me! — Reply to this email directly, view it on GitHub <#2701 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH2WKOZ53CHAUV6BFZJXCATZRT4IDAVCNFSM6AAAAABMKAEAG2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJSGAYTONRSHE> . You are receiving this because you commented.Message ID: ***@***.***>

[stepfunction83]

With a 24GB card, I run out of VRAM after about 30 or so training steps.

[jpXerxes]

With a 24GB card, I run out of VRAM after about 30 or so training steps.
Hmm. Same 24GB here (4090) and it ran fine through 320 steps. ai-toolkit talks about using lowvram when you "only" have 24GB and some is being used for the display, but like I said it ran fine here.

[pAInCREAT0R]

A lot of recent postings (external) recommend using non distilled versions of FLUX for finetuning. I know when I try to use them instead of Flux1_dev, it errors out. (I believe this is related to how the metadata is structured, but I could be wrong.) Will we be able to support the training for the non distilled model? I have been trying train the Flux_Dev checkpoint - it has been successful for likenesses of individual - but full finetuning of multiple concepts has been failures so far. The model is so resilient and responsive to LoRA training, but seems to breakdown quickly under dreambooth finetuning. I am hoping having the distillation removed will be more predictable. (Sorry if I am misstating any of this as I am an avid hobbyist but very new to the more technical side.)

[dsienra]

A lot of recent postings (external) recommend using non distilled versions of FLUX for finetuning. I know when I try to use them instead of Flux1_dev, it errors out. (I believe this is related to how the metadata is structured, but I could be wrong.) Will we be able to support the training for the non distilled model? I have been trying train the Flux_Dev checkpoint - it has been successful for likenesses of individual - but full finetuning of multiple concepts has been failures so far. The model is so resilient and responsive to LoRA training, but seems to breakdown quickly under dreambooth finetuning. I am hoping having the distillation removed will be more predictable. (Sorry if I am misstating any of this as I am an avid hobbyist but very new to the more technical side.)

Flux dev is distilled, a non distilled version is not available, Flux pro is the not distilled one but the weights were not released publicly, there are some ideas to undo the distillation process at least in the human class to avoid class bleeding, training the model Flux dev with a diverse dataset of people of all races genders and ages and then on top of this finetune you train your new subjects, this was not tested but is promising that the model will behave more like a non distilled model when training people, but doing this first finetune will require time and resources.

[jpXerxes]

Flux dev is distilled, a non distilled version is not available,

Hmm. I've had good results with dropping this in, in place of regular flux dev and no other changes: https://civitai.com/models/812892/flux1-dev2pro

[dsienra]

Flux dev is distilled, a non distilled version is not available,

Hmm. I've had good results with dropping this in, in place of regular flux dev and no other changes: https://civitai.com/models/812892/

some people said that this model is probably fake, the original version was published on hugging face more than a month ago and it claimes that were trained on 3 million images ir will have to take a lot of time and resources, but if you are getting good results who knows... I will give it a try

[dsienra]

Flux dev is distilled, a non distilled version is not available,

Hmm. I've had good results with dropping this in, in place of regular flux dev and no other changes: https://civitai.com/models/812892/

some people said that this model is probably fake, the original version was published on hugging face more than a month ago and it claimes that were trained on 3 million images ir will have to take a lot of time and resources, but if you are getting good results who knows... I will give it a try

I will try to train many people at the same time and see if still bleeds...

[jpXerxes]

if you are getting good results who knows... I will give it a try

"good" when it comes to these things is subjective. My opinion is that faces are more realistic. They do caution to not apply loras produced with this version back with the version, but rather use distilled.

[jpXerxes]

A lot of recent postings (external) recommend using non distilled versions of FLUX for finetuning. I know when I try to use them instead of Flux1_dev, it errors out.

Did you try the one I pointed to, or another? I tried using the FP8 version, but that errored out.

[dsienra]

if you are getting good results who knows... I will give it a try

"good" when it comes to these things is subjective. My opinion is that faces are more realistic. They do caution to not apply loras produced with this version back with the version, but rather use distilled.

Is not just that the faces looks realistic, the problem with distilled models is that they are unable to learn many people of the same class without mixing them together, flux dev suffers of bleeding and catastrophic forgetting.

[pAInCREAT0R]

A lot of recent postings (external) recommend using non distilled versions of FLUX for finetuning. I know when I try to use them instead of Flux1_dev, it errors out. (I believe this is related to how the metadata is structured, but I could be wrong.) Will we be able to support the training for the non distilled model? I have been trying train the Flux_Dev checkpoint - it has been successful for likenesses of individual - but full finetuning of multiple concepts has been failures so far. The model is so resilient and responsive to LoRA training, but seems to breakdown quickly under dreambooth finetuning. I am hoping having the distillation removed will be more predictable. (Sorry if I am misstating any of this as I am an avid hobbyist but very new to the more technical side.)

Flux dev is distilled, a non distilled version is not available, Flux pro is the not distilled one but the weights were not released publicly, there are some ideas to undo the distillation process at least in the human class to avoid class bleeding, training the model Flux dev with a diverse dataset of people of all races genders and ages and then on top of this finetune you train your new subjects, this was not tested but is promising that the model will behave more like a non distilled model when training people, but doing this first finetune will require time and resources.

https://www.reddit.com/r/StableDiffusion/comments/1fuukwz/fluxdevdedistill_an_undistilled_version_of_flux/ I am not saying if they are legit, but there are two models claiming to have removed the distillation being promoted on Reddit. One full and the other FP8 I believe. I know Furkan has posted about them as well.

[dsienra]

A lot of recent postings (external) recommend using non distilled versions of FLUX for finetuning. I know when I try to use them instead of Flux1_dev, it errors out. (I believe this is related to how the metadata is structured, but I could be wrong.) Will we be able to support the training for the non distilled model? I have been trying train the Flux_Dev checkpoint - it has been successful for likenesses of individual - but full finetuning of multiple concepts has been failures so far. The model is so resilient and responsive to LoRA training, but seems to breakdown quickly under dreambooth finetuning. I am hoping having the distillation removed will be more predictable. (Sorry if I am misstating any of this as I am an avid hobbyist but very new to the more technical side.)

Flux dev is distilled, a non distilled version is not available, Flux pro is the not distilled one but the weights were not released publicly, there are some ideas to undo the distillation process at least in the human class to avoid class bleeding, training the model Flux dev with a diverse dataset of people of all races genders and ages and then on top of this finetune you train your new subjects, this was not tested but is promising that the model will behave more like a non distilled model when training people, but doing this first finetune will require time and resources.

https://www.reddit.com/r/StableDiffusion/comments/1fuukwz/fluxdevdedistill_an_undistilled_version_of_flux/ I am not saying if they are legit, but there are two models claiming to have removed the distillation being promoted on Reddit. One full and the other FP8 I believe. I know Furkan has posted about them as well.

Ok, good news if it works, I will give it a try

[FurkanGozukara]

i think real distilled will be ostris one https://huggingface.co/ostris/OpenFLUX.1

but i think too early atm

[pAInCREAT0R]

Can you train a LoRA in the Khoya GUI now on a checkpoint made with Ostris LoRA's merged into them? I was not sure if that was working or if it errored out like the finetuning does. I have it working in AIToolkit, but much prefer the GUI here for larger projects.

[ChristianMayer]

Not at the moment. I have not looked at the lycoris code in a long time... Might have to look into it.

The recent 3.1.0 release (02.10.24) has Flux support as stated in https://github.com/KohakuBlueleaf/LyCORIS/blob/main/Change.md

So it would be great to use the GUI to train LoKR and DoRA

[FurkanGozukara]

Not at the moment. I have not looked at the lycoris code in a long time... Might have to look into it.

The recent 3.1.0 release (02.10.24) has Flux support as stated in https://github.com/KohakuBlueleaf/LyCORIS/blob/main/Change.md

So it would be great to use the GUI to train LoKR and DoRA

awesome

DoRA better than LokR?

[ChristianMayer]

When we can train both we can compare.

nVidia had a paper where they discussed that DoRA should/could/will be the future replacement for LoRA as it is much better and has no negatives with it.
But every paper is claiming that about their innovation.

[bmaltais]

Not at the moment. I have not looked at the lycoris code in a long time... Might have to look into it.

The recent 3.1.0 release (02.10.24) has Flux support as stated in https://github.com/KohakuBlueleaf/LyCORIS/blob/main/Change.md

So it would be great to use the GUI to train LoKR and DoRA

Last time I attempted to update to the Lycoris 3.x version, most Lycoris-based training modules broke. It doesn’t seem to be backward compatible, and it will require substantial effort to use in the GUI. I’ve pushed that aside because I didn’t want to invest the time to make it work. However, I might consider giving it a try as part of the sd3 branch.

[bmaltais]

Not at the moment. I have not looked at the lycoris code in a long time... Might have to look into it.

The recent 3.1.0 release (02.10.24) has Flux support as stated in https://github.com/KohakuBlueleaf/LyCORIS/blob/main/Change.md

So it would be great to use the GUI to train LoKR and DoRA

OK, I updated the requirements file and re-aligned the GUI to match the documented options parameter support... hope this allow you to train a Flux.1 LoRA using the Lycoris method.

[diodiogod]

Can you train a LoRA in the Khoya GUI now on a checkpoint made with Ostris LoRA's merged into them? I was not sure if that was working or if it errored out like the finetuning does. I have it working in AIToolkit, but much prefer the GUI here for larger projects.

I have this exact same problem. I managed to merge 3 loras into a checkpoint (with adjusted block weights) using ComfyUI, I can use the checkpoint, but I cannot extract a LoRa from it or use it to train another LoRA on Kohya... I think merging the diffuser format breaks the checkpoint in kohya. It's funny that it can be used normally on Comfy or Forge though.

[leporel]

OK, I updated the requirements file and re-aligned the GUI to match the documented options parameter support... hope this allow you to train a Flux.1 LoRA using the Lycoris method.

2d_toska_v1.5_lycoris_dylora.json

if start train with checked Flux.1 checkbox i have error
09:37:55-984016 ERROR LoRA type must be set to 'Flux1' or 'Flux1 OFT' if Flux1 checkbox is checked.

if uncheck i have this error

log

INFO     loading model for process 0/1                                                            train_util.py:5245
INFO     load StableDiffusion checkpoint:                                                         train_util.py:5201
                           G:/StabilityMatrix/Train/flux1-dev-fp8-e4m3fn.safetensors
Traceback (most recent call last):
File "G:\StabilityMatrix\Data\Packages\kohya_ss\sd-scripts\train_network.py", line 1454, in <module>
  trainer.train(args)
File "G:\StabilityMatrix\Data\Packages\kohya_ss\sd-scripts\train_network.py", line 354, in train
  model_version, text_encoder, vae, unet = self.load_target_model(args, weight_dtype, accelerator)
File "G:\StabilityMatrix\Data\Packages\kohya_ss\sd-scripts\train_network.py", line 102, in load_target_model
  text_encoder, vae, unet, _ = train_util.load_target_model(args, weight_dtype, accelerator)
File "G:\StabilityMatrix\Data\Packages\kohya_ss\sd-scripts\library\train_util.py", line 5247, in load_target_model
  text_encoder, vae, unet, load_stable_diffusion_format = _load_target_model(
File "G:\StabilityMatrix\Data\Packages\kohya_ss\sd-scripts\library\train_util.py", line 5202, in _load_target_model
  text_encoder, vae, unet = model_util.load_models_from_stable_diffusion_checkpoint(
File "G:\StabilityMatrix\Data\Packages\kohya_ss\sd-scripts\library\model_util.py", line 1005, in load_models_from_stable_diffusion_checkpoint
  converted_unet_checkpoint = convert_ldm_unet_checkpoint(v2, state_dict, unet_config)
File "G:\StabilityMatrix\Data\Packages\kohya_ss\sd-scripts\library\model_util.py", line 267, in convert_ldm_unet_checkpoint
  new_checkpoint["time_embedding.linear_1.weight"] = unet_state_dict["time_embed.0.weight"]
KeyError: 'time_embed.0.weight'
Traceback (most recent call last):
File "C:\Users\lepa\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
  return _run_code(code, main_globals, None,
File "C:\Users\lepa\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
  exec(code, run_globals)
File "G:\StabilityMatrix\Data\Packages\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in <module>
File "G:\StabilityMatrix\Data\Packages\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main
  args.func(args)
File "G:\StabilityMatrix\Data\Packages\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command
  simple_launcher(args)
File "G:\StabilityMatrix\Data\Packages\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher
  raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['G:\\StabilityMatrix\\Data\\Packages\\kohya_ss\\venv\\Scripts\\python.exe', 'G:/StabilityMatrix/Data/Packages/kohya_ss/sd-scripts/train_network.py', '--config_file', 'G:/StabilityMatrix/Train/DataSets/model/config_lora-20241017-094443.toml']' returned non-zero exit status 1.
09:44:52-459238 INFO     Training has ended.

maybe i use wrong settings? can someone make preset for flux LyCoris ?

[bmaltais]

@leporel RIght... I had that there when I tried to make sure users could only select FLux.1 compatible models... Let me add the ones for LyCORIS to the list...

[bmaltais]

@leporel OK, Try pulling the latest commit. Should allow to use LyCORIS LoRA types with Flux.1... hope that work.

[Pashahlis]

I can confirm that it works now.

[DarkViewAI]

@leporel OK, Try pulling the latest commit. Should allow to use LyCORIS LoRA types with Flux.1... hope that work.

does it only work with attn-only and not full?

cant seem to get full to work

[bmaltais]

I can confirm that it works now.

Were you able to obtain a functional LoRA from it?

[Pashahlis]

@leporel OK, Try pulling the latest commit. Should allow to use LyCORIS LoRA types with Flux.1... hope that work.

does it only work with attn-only and not full?

cant seem to get full to work

Full worked for me.

[Pashahlis]

I can confirm that it works now.

Were you able to obtain a functional LoRA from it?

Yes.

[Pashahlis]

Maybe I should add that I only tried Lycoris LoCon so far. Trying LoHa now.

[Pashahlis]

LoHa works fine with full, too. Just increases "training intensity" a lot so you gotta lower rank/alpha and/or learning rates accordingly to not overtrain.

[leporel]

LoHa works fine with full, too. Just increases "training intensity" a lot so you gotta lower rank/alpha and/or learning rates accordingly to not overtrain.

can you post your config file (preset) ?
I have error AttributeError: 'LycorisNetworkKohya' object has no attribute 'train_blocks' and I don't figure out where is problem
^ error occurs when select "Split mode"
suddenly i cant train on 12gb vram in this case, but anyway it seems train DyLora not work and throw error RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x3072 and 1152x16)

[uzitotte]

I git cloned to linux and ran requirements but I cant spawn the Flux tickbox in the Lora tab even after chosing the model.

Am I'm missing something? SDXL training works great..