Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix potential autocast NaNs in image upscale #14809

Merged

Conversation

Cyberbeing
Copy link
Contributor

Description

Bugfix for potential autocast NaNs in image upscale split off from Pull Request #14794

Discussion in that other pull request now confirms this works for others for preventing the black screen NaNs with Float32 upscaler model architectures [1] [2]. The performance impacts for Float16 converted models was also recently confirmed to be minimal after some recent changes [3].

Float32 upscaler models may now be slower, since they will be processed more than before in Float32 instead of autocast, but this seems necessary to fix the NaNs. In the future, I imagine this performance impact could be lifted by supporting BFloat16 upscaling, which Spandrel seems to support for every model architecture unlike Float16.

Spandrel Arch supporting Float32/BFloat16 only: CodeFormer, DAT, GFPGAN, GRL, HAT, SRFormer, SwinIR

Spandrel Arch supporting Float32/Float16/BFloat16: Compact, ESRGAN, RealESRGAN, OmniSR, SPAN
Note: Float16 support for ESRGAN and others requires the additional changes in #14794. Currently of the currently supported model types, prefer_half which triggers float16 model conversion only seems to be enabled for realesrgan_model.py but not esrgan_model.py

Checklist:

@AUTOMATIC1111 AUTOMATIC1111 merged commit 652a7bb into AUTOMATIC1111:dev Jan 31, 2024
3 checks passed
@Cyberbeing Cyberbeing deleted the fix_upscaler_autocast_nans branch January 31, 2024 22:08
@Cyberbeing Cyberbeing mentioned this pull request Jan 31, 2024
4 tasks
@w-e-w w-e-w mentioned this pull request Feb 17, 2024
@pawel665j pawel665j mentioned this pull request Apr 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants