Fix potential autocast NaNs in image upscale #14809
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Bugfix for potential autocast NaNs in image upscale split off from Pull Request #14794
Discussion in that other pull request now confirms this works for others for preventing the black screen NaNs with Float32 upscaler model architectures [1] [2]. The performance impacts for Float16 converted models was also recently confirmed to be minimal after some recent changes [3].
Float32 upscaler models may now be slower, since they will be processed more than before in Float32 instead of autocast, but this seems necessary to fix the NaNs. In the future, I imagine this performance impact could be lifted by supporting BFloat16 upscaling, which Spandrel seems to support for every model architecture unlike Float16.
Spandrel Arch supporting Float32/BFloat16 only: CodeFormer, DAT, GFPGAN, GRL, HAT, SRFormer, SwinIR
Spandrel Arch supporting Float32/Float16/BFloat16: Compact, ESRGAN, RealESRGAN, OmniSR, SPAN
Note: Float16 support for ESRGAN and others requires the additional changes in #14794. Currently of the currently supported model types, prefer_half which triggers float16 model conversion only seems to be enabled for realesrgan_model.py but not esrgan_model.py
[Bug]: DAT upscaler is not working in hires fix. #14710 (comment)
Support more Upscalers #14794
Checklist: