Normalize only nonzero normals for mikktspace normal maps #10905
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Objective
Fixes #5891.
For mikktspace normal maps, normals must be renormalized in vertex shaders to match the way mikktspace bakes vertex tangents and normal maps so that the exact inverse process is applied when shading.
However, for invalid normals like
vec3<f32>(0.0, 0.0, 0.0)
, this normalization causes NaN values, and because it's in the vertex shader, it affects the entire triangle and causes it to be shaded as black:A cone with a tip that has a vertex normal of [0, 0, 0], causing the mesh to be shaded as black.
In some cases, normals of zero are actually useful. For example, a smoothly shaded cone without creases requires the apex vertex normal to be zero, because there is no singular normal that works correctly, so the apex shouldn't contribute to the overall shading. Duplicate vertices for the apex fix some shading issues, but it causes visible creases and is more expensive. See #5891 and #10298 for more details.
For correctly shaded cones and other similar low-density shapes with sharp tips, vertex normals of zero can not be normalized in the vertex shader.
Solution
Only normalize the vertex normals and tangents in the vertex shader if the normal isn't [0, 0, 0]. This way, mikktspace normal maps should still work for everything except the zero normals, and the zero normals will only be normalized in the fragment shader.
This allows us to render cones correctly:
Notice how there is still a weird shadow banding effect in one area. I noticed that it can be fixed by normalizing here, which produces a perfectly smooth cone without duplicate vertices:
I didn't add this change yet, because it seems a bit arbitrary. I can add it here if that'd be useful or make another PR though.