Min-SNR Gamma: correct the fix for SNR weighted loss in v-prediction … #5238
Conversation
…by adding 1 to SNR rather than the resulting loss weights
|
Training two models from scratch to test this one, concurrently. They are using the exact same hyperparameters, with the exception of min-SNR gamma. With SNR gamma:
Without SNR gamma:
|
|
The loss function is very similar but, it has about .1 lower on avg at each step for the gamma model vs the vanilla model. The results themselves seem more substantially-improved than .1 loss would usually indicate. With gamma:
Without gamma:
|
|
the
|
|
just for completeness, at 1k steps on both models, the gap is really beginning to widen, where the min-SNR gamma model seriously pulls ahead in terms of representing details and coherence.
|
|
@bghira Doesn't this change reintroduce the division by zero issue when using zero terminal SNR + epsilon prediction? As is, the weight will be |
|
in my opinion, you shouldn't be combining zero terminal SNR with epsilon anyway. i'm sure there's an additional fix that can be applied, but this trainer uses the simplest implementations of the logic from the research paper. using epsilon would be out of scope for that? @sayakpaul thoughts on that? |
|
Yeah I agree. |
huggingface#5238) Min-SNR Gamma: correct the fix for SNR weighted loss in v-prediction by adding 1 to SNR rather than the resulting loss weights Co-authored-by: bghira <[email protected]> Co-authored-by: Sayak Paul <[email protected]>
…by adding 1 to SNR rather than the resulting loss weights
What does this PR do?
Fixes issue as mentioned by @parlance-zz here.
Sorry for all the back and forth on this stuff, I did about 6000 steps of training with the previous fix and all was well. If this works better, awesome!