Fix min-snr-gamma for v-prediction and ZSNR. #934
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Based on the discussion in #673
Matches the reference implementation from the paper's author.
Note: With this PR, you should use
--min_snr_gamma
without--scale_v_pred_loss_like_noise_pred
The old implementation with
apply_snr_weight
andscale_v_prediction_loss_like_noise_prediction
splits the calculation into two steps:The problem here is that with
--zero_terminal_snr
it is possible for the first step to divide by zero.This PR simplifies the weight calculation to
min(snr, gamma)/(snr+1)
to avoid dividing by zero.This makes
scale_v_prediction_loss_like_noise_prediction
unnecessary, but I did not remove it yet to avoid breaking existing workflows and GUIs. Maybe a deprecation notice should be added to it?