Add support for infinite projection matrix

[Flarkk]

Closes godotengine/godot-proposals#10515.
Fixes #55070

What is it about ?

This PR ensures that :

• class Projection returns a well-formed far plane when the matrix has the form of an infinite projection
• the rendering pipeline works as expected when the camera far plane is very far away

Without this PR, when the projection matrix is infinite, Projection generate zero vectors for the far plane normal, and 0 as z-far distance. This raise errors at different places in the rendering logic (culling, shadow mapping, GI, fog ...) and ultimately prevents anything to be rendered.

With this PR, Projection returns z-far distance as MAX_REAL_T and the far plane's normal as Vector3(0, 0, -1) when the projection matrix has the form of an infinite projection.

Why it does matter ?

Godot 4.3 brought reverse z-buffer which unlocks rendering with very far apart near and far planes, with very limited risks of z-fighting in normal conditions.
However, in practice the usage of very far apart near and far planes is still not supported because it makes rendering fail for the reason described above.

This PR ultimately untaps the potential of natively supporting very deep scenes in Godot, without having to workaround it with multiple nested/layered cameras. This is especially useful for terrain rendering and space simulations.

Under which conditions is an infinite projection matrix generated ?

The infinite form of a projection matrix is obtained when zfar tends towards infinity.
Due to numerical precision limits, in practice it's enough that znear and zfar values are far apart enough to obtain an infinite projection matrix.
I experienced that beyond a difference of 1e7 ~ 1e8 between the near and far distances, the matrix is generated infinite.

The below collapsed sections show how Perspective, Frustum and Orthogonal projections react to far apart znear and zfar when constructed from camera attributes. In a nutshell :

• For Perspective and Frustum, columns[2][2] = -1 and columns[3][2] = -2 * p_z_near
• For Orthogonal, columns[3][2] = -1

Expand details

Perspective projection

From Projection::set_perspective():

deltaZ = p_z_far - p_z_near; // becomes p_z_far
// [...]
columns[0][0] = cotangent / p_aspect;
columns[1][1] = cotangent;
columns[2][2] = -(p_z_far + p_z_near) / deltaZ; // becomes -1
columns[2][3] = -1;
columns[3][2] = -2 * p_z_near * p_z_far / deltaZ; // becomes -2 * p_z_near

Frustum projection

From Projection::set_frustum(), rewritten for clarity:

columns[0][0] = 2 * p_near / (p_right - p_left);
columns[1][1] = 2 * p_near / (p_top - p_bottom);
columns[2][0] = (p_right + p_left) / (p_right - p_left);
columns[2][1] = (p_top + p_bottom) / (p_top - p_bottom);
columns[2][2] = -(p_far + p_near) / (p_far - p_near); // becomes -1
columns[2][3] = -1;
columns[3][2] = -2 * p_far * p_near / (p_far - p_near); // becomes -2 * p_z_near

Orthogonal projection

From Projection::set_orthogonal()

columns[0][0] = 2.0 / (p_right - p_left);
columns[1][1] = 2.0 / (p_top - p_bottom);
columns[2][2] = -2.0 / (p_zfar - p_znear); // becomes -2.0 / p_zfar
columns[3][0] = -((p_right + p_left) / (p_right - p_left));
columns[3][1] = -((p_top + p_bottom) / (p_top - p_bottom));
columns[3][2] = -((p_zfar + p_znear) / (p_zfar - p_znear)); // becomes -1
columns[3][3] = 1.0;

Why an infinite Perspective and Frustum projection matrices breaks zfar and far plane extraction ?

User camera settings are transformed into a projection matrix stored in a Projection object.
When needed, the rendering server extracts back the frustum planes and distances from the Projection object.
This happens in one of these 4 functions :

• Projection::get_projection_planes()
• Projection::get_projection_plane()
• Projection::get_z_far()
• Projection::get_far_plane_half_extents()

When the projection matrix degenerates to its infinite form due to numerical precision effects, it becomes mathematically impossible to extract the far plane normal and intercept from it.
When performed by one of those functions, the calculation gives 0 (zero vector as the normal, and 0 intercept).
This is the relevant code excerpt taken from Projection::get_projection_plane() and rewritten for clarity (the logic is the same for other functions) :

// This initializes a plane with a zero vector as its normal (first 3 arguments)
Plane new_plane = Plane(columns[0][3] - columns[0][2], // = 0 in all cases
		columns[1][3] - columns[1][2], // = 0 in all cases
		columns[2][3] - columns[2][2], // = 0 for infinite Perspective and Frustum projection
		columns[3][3] - columns[3][2]);

new_plane.normal = -new_plane.normal;
new_plane.normalize(); // returns a zero plane when the normal is a zero vector
return new_plane;

Orthogonal projections aren't affected because columns[2][2] = -2.0 / (p_zfar - p_znear) which becomes -2.0 / p_zfar when z_far is big, which is still a (small) finite value. This is under the assumption that p_zfar is not set to INFINITY by the user, which is currently not possible through the editor, but possible with code.

TODOs

• Detect infinite projections and return correct far plane (normal = Vector3(0, 0, -1) and intercept = MAX_REAL_T)
• List usages of extracted zfar value and check whether MAX_REAL_T breaks any further logic
  • Forward renderers (clustered and mobile)
  • Compatibility renderer
  • Sky
  • GI
  • Fog
  • SS effects
  • Debug effects
  • Scene renderer
  • Cluster builder
  • Scene culling
  • Raycast occlusion culling
  • Light culler
  • Scene renderer
• Fix any broken logic with zfar = MAX_REAL_T
  • Forward renderers (clustered and mobile)
  • Compatibility renderer
• Write unit tests with infinite far planes
  • Camera3D
  • XRCamera3D
  • Plane
• Add other features using zfar / far plane (like GI and light culling) in the demo project

Infinite zfar impact analysis

The following collapsed sections document the behaviors of the various impacted steps of the rendering pipeline when the zfar value extracted from the projection matrix is MAX_REAL_T.

✅ means the behaviour is likely fine.
⚠️ means there is likely a problem. Further investigation is needed.

Note that for now this assessment is only based code inspection.
Proper testing will be needed in all cases.

Show detailed analysis

RenderForwardClustered (+ RasterizerSceneGLES3)

RenderForwardClustered::_fill_render_list()
RasterizerSceneGLES3::_fill_render_list():

• ✅ All surfaces are set to depth_layer 0. OK since depth_layer seems not to be used as a sort key

RenderForwardClustered and RenderForwardClusteredMobile (+ RasterizerSceneGLES3)

RenderForwardClustered::_render_particle_collider_heightfield()
RenderForwardClusteredMobile::_render_particle_collider_heightfield()
RasterizerSceneGLES3::render_particle_collider_heightfield()

• ⚠️ scene_data has z_far == MAX_REAL_T which leads the ubo to have z_far = MAX_REAL_T too

SkyRD (+ RasterizerSceneGLES3)

SkyRD::setup_sky():

• ✅The sky scene UBO has z_far = MAX_REAL_T, which leads the sky shader to have z_far = MAX_REAL_T. This value is not used anywhere in the glsl file though.

SkyRD::update_res_buffers()
SkyRD::draw_sky():
RasterizerSceneGLES3::_draw_sky()

• ✅ If the sky has a custom fov, build an infinite perspective projection with the custom fov. Otherwise use the camera's projection, which is infinite too

GI

GI::process_gi():

• ⚠️ GI shader gets zfar = MAX_REAL_T via the push constants. This makes the glsl function reconstruct_position() return vec2(NaN) in all cases, which makes process_gi() run on NaN vertices, which likely gives NaN in the end too (not checked)

Fog

Fog::volumetric_fog_update() :

• ✅ For each fog volume : Fog shader gets zfar = MAX_REAL_T via the UBO , but this value is not used anywhere in the glsl file

• ⚠️ The Fog process shader gets fog_frustum_size_begin = fog_frustum_size_end = frustum_near_size as well as zfar = MAX_REAL_T via the UBO. I haven't checked the effets yet but it's lilkely the fog is not rendered correctly

SSEffects

SSEffects::downsample_depth() :

• ✅ The downsampling shader gets zfar = MAX_REAL_T via the push constants only in Orthogonal mode, which is not affected by the infinite matrix issue

SSEffects::screen_space_indirect_lighting :

• ⚠️ The ssil shader gets zfar = MAX_REAL_T via push constants which makes all reprojected samples in last frame's coordinates lie on the near plane. I'm not sure what are the impacts for now.

SSEffects::screen_space_reflection :

• ⚠️ The ssr scale shader gets camera_z_far = MAX_REAL_T via push constants which generates NaNs in depth values in two places (1 2). Impacts not checked, but probably not good

• ⚠️ The ssr shader gets camera_z_far = MAX_REAL_T via push constants which generates infinite ray ends and NaN depths. Impacts not checked, but probably not good

SSEffects::sub_surface_scattering :

• ⚠️ The sss shader gets camera_z_far = MAX_REAL_T via push constants which generates NaNs in depth values. Impacts not checked, but probably not good

DebugEffects

DebugEffects::draw_shadow_frustum():

• ⚠️ Projections of all shadow splits likely have both near and far set as MAX_REAL_T. Impacts not checked, but probably not good

RendererSceneRenderRD (+ RasterizerSceneGLES3)

RendererSceneRenderRD::render_scene()
RasterizerSceneGLES3::render_scene():

• ⚠️ RenderSceneDataRD UBO has zfar = MAX_REAL_T which has several effects in the scene shader : fog mip level is always 1, multiview cluster is always 0, and interpolated vertex for dual paraboloid always lies on the near plane

RendererSceneRenderRD::render_scene()

• ⚠️ FSR2 might emit warnings as it implements infinite far with the maximum representable float instead of infinity
• ✅ TAA::process() and TAA::resolve() take z-depth and z-near as parameters but aren't used in the functions bodies

ClusterBuilderRD

ClusterBuilderRD::begin():

• ✅ ClusterBuilderRD::add_box() and ClusterBuilderRD::add_light() : render elements never touch the far plane
• ⚠️ The state_uniform buffer gets inv_z = 0, which makes the z cluster be always 0 in the cluster_render shader
• ⚠️ The debug shader gets zfar == MAX_REAL_T through push constants which generates NaN depth calculation and likely make it output a solid color

RendererSceneCull

RendererSceneCull::_light_instance_setup_directional_shadow():

• ⚠️ Projections of all shadow splits but the last one have both near and far set as camera znear. Impacts not checked, but probably not good. Probably what causes directional light shadows not working in large world #81877

RendererSceneCull::_light_instance_update_shadow():

• ✅ Convex queries should work correctly with infinite far plane

RendererSceneCull::_scene_cull():

• ✅ frustum culling should work correctly with infinite far plane.

RaycastOcclusionCull::RaycastHZBuffer

RaycastOcclusionCull::RaycastHZBuffer::update_camera_rays():

• ⚠️ Camera rays get zfar = MAX_REAL_T. Might have impacts on Embree logic
• ⚠️ Mips are all set to MAX_REAL_T too

RenderingLightCuller

RenderingLightCuller::_add_light_camera_planes
RenderingLightCuller::add_light_camera_planes_directional
RenderingLightCuller::cull_regular_light:

• ✅ Far plane has infinite intercept. Light culling should be performed correctly anyway

RendererSceneRender::CameraData

RendererSceneRender::CameraData::set_multiview_camera():

• ✅ The combined projection matrix should be fine with infinite far

Test project

PR95944.zip

Camera has near = 0.05 and far ~= 1e20, perspective projection.
Pink sphere is at z ~= -1e20
Blue sphere is at z = -1e15
Green sphere is at z = -1e10
Orange sphere is at z = -1e5
Black sphere is at z = -1

Without this PR :

With this PR :

[tetrapod00]

Does this introduce another change in behavior when working with clip space in shaders like reverse-z did, like e.g. reconstructing position from depth? Or does the math work out such that shaders are unchanged? I don't see any GLSL in the changed files.

[Flarkk]

No, there is no impact on shaders.
In fact, the only thing this PR changes is how the frustum planes (and specifically the far plane) are extracted from the projection matrix. The matrix itself is passed through unchanged as before to the rest of the pipeline, including the GPU parts.
Frustum planes are extracted and used in a handful of culling operations, on the CPU, mainly geometry culling and light culling.
You can find them back by searching calls to Projection::get_projection_planes() throughout the code.
This is really the only function affected by the PR, speaking of the core computational logic.

[Flarkk]

Stumbled upon a few other places in Projection that need the fix on z plane calculation (other than Projection::get_projection_planes()).

Some of those functions are not used anywhere in the code but could still be called in user modules or custom builds :

• Projection::get_projection_plane()
• Projection::get_z_far()
• Projection::get_far_plane_half_extents()

I just updated the PR.

[Flarkk]

Notes from the rendering weekly meeting :

is it fine to “detect” infinite matrices from their values right into Projection, instead of passing through a flag set by Camera3D based on far value ? (I’ve seen a “is_orthogonal” flag passed through whereas orthogonality could have been likewise detected from the matrix components themselves)

Yes, detecting infinite far is fine, but the fact that the projection uses an infinite far should propagate to further steps in rendering (i.e. the construction of the projection matrix + projection for screen space effects)

any further test project needed to bullet proof the fix ? (thinking of light culling, fog rendering, gi, and other stuff that uses far plane extraction)

Extensive testing will be needed. I don't think we have test coverage over this yet. So probably need to write code tests as well as a graphical demo for comparison

any other changes you may have in mind to fully unlock infinite far planes ? (thinking of 1 minor improvement : remove far distance capping in editor - but there might be more)

This is a good start, but the support would need to reach through the rendering pipeline to anything that is touched by the far plane. We rely on the far distance in many places in the renderer, all of these need to be checked

[roalyr]

I am sorry, I can not help with tests on this one after all. Too much work and life issues now.

[Flarkk]

Just rewrote the description of this PR to clarify what it is about exactly. Please refer to it as it may help with the below.

To the points raised during the rendering meeting (assuming @clayjohn you made most of them) :

Yes, detecting infinite far is fine, but the fact that the projection uses an infinite far should propagate to further steps in rendering (i.e. the construction of the projection matrix + projection for screen space effects)

This is a good start, but the support would need to reach through the rendering pipeline to anything that is touched by the far plane. We rely on the far distance in many places in the renderer, all of these need to be checked

This propagates already by design : the current workflow is basically user camera settings -1-> baked into a matrix in the Projection object -2-> far plane extraction from the matrix.

The only thing this PR changes is which far plane is returned at step 2. when the matrix is detected infinite, instead of a zero-plane (zero normal and zero intercept) which raise errors for obvious reasons. What it does right now is that it returns Vector3(0, 0, -1) as the normal and INFINITY as the intercept (zfar value).

I believe what's in question is not the propagation, but rather whether Vector3(0, 0, -1) and INFINITY are good choices and don't mess things up down the line :

• I'm pretty confident for Vector3(0, 0, -1) as the far plane is always perpendicular to Z in all settings (Perspective, Frustum, Orthogonal). This might not be the case anymore with user-provided projection matrices though, but I'm wondering if having non Z-perpendicular infinite far planes is a use case at all.
• INFINITY as the intercept raises more questions in my opinion : is there any place this would generate wrong results or raise errors ? My assumption is no, and the attached demo project tends to demonstrate it, but I agree more torough testing is required here

Extensive testing will be needed. I don't think we have test coverage over this yet. So probably need to write code tests as well as a graphical demo for comparison

I agree it's needed for intercept = INFINITY (see above answer). Also, the attached graphical demo covers the geometry culling part, but not all functionalities relying of zfar / far plane, like GI or light culling.

I added a todolist in the PR description to track those additional steps down.

[Flarkk]

@clayjohn I've started to document each impact across the rendering pipeline. See section Infinite zfar impact analysis in the PR description.
Can you take a quick look and tell me if this is on the right track ?