[Clarification] LORA layer scaling lr

[SavvaI]

I am apologising in advance if I misunderstood something.
My issue comes from the observation, that when I raise the lora rank (4 -> 512) the observable effective learning rate (difference between the sampled images through training) drastically drops.
So I come to the source code and see https://github.com/kohya-ss/sd-scripts/blob/c93cbbc373daff7827395b6ca5bde91733890722/networks/lora.py#L52
self.scale = alpha / self.lora_dim
In my understanding, the right way to implement the equalised learning rate https://arxiv.org/abs/1812.04948 , should be the following:
self.scale = alpha / (in_dim**0.5) / (self.lora_dim**0.5)
(in_dim**0.5) divider for the down_sample layer and (self.lora_dim**0.5) for the up_sample layer.
Thank you.

[kohya-ss]

The alpha is based on the following paper and the repo.
https://arxiv.org/abs/2106.09685
https://github.com/microsoft/LoRA

And the introducing of the alpha was discussed this issue:
kohya-ss/sd-webui-additional-networks#49

I am not good at math, so I'm not sure these are better or not than your description, but I hope they will help for clarification.