Fix regression with FLUX diffusers LoRA models

[RyanJDick]

Summary

Fixes a regression in support for FLUX diffusers LoRA models (introduced in #6967).

Related Issues / Discussions

Closes #6996

QA Instructions

I ran the following manual tests:

• Non-quantized transformer
  • diffusers LoRA
  • kohya LoRA with transformer layers only
  • kohya LoRA with transformer layers and CLIP layers
• Quantized tranformer (BnB NF4)
  • diffusers LoRA
  • kohya LoRA with transformer layers only
  • kohya LoRA with transformer layers and CLIP layers

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable)
• Documentation added / updated (if applicable)