Initial TAA implementation

[JFonS]

Initial TAA support based on the implementation in Spartan Engine.

Motion vectors are correctly generated for camera and mesh movement, but there is no support for other things like particles or skinned mesh deformations.

Here's a still image comparison: https://imgsli.com/MTA5MDA5

Bugsquad edit: This closes godotengine/godot-proposals#3401.

TAA Enabled

TAA Disabled

@Calinou You mentioned adding a project setting for the TAA blend factor. I played around with it for a bit, but I didn't see much use for it. Here's the patch for my changes with a hard-coded blend factor, if you want to take a look yourself. Feel free to open a PR if you think users could benefit from such a project setting:

Blend factor patch

diff --git a/servers/rendering/renderer_rd/effects_rd.cpp b/servers/rendering/renderer_rd/effects_rd.cpp
index 70a160eeb9..b7d3b5f76a 100644
--- a/servers/rendering/renderer_rd/effects_rd.cpp
+++ b/servers/rendering/renderer_rd/effects_rd.cpp
@@ -252,11 +252,12 @@ void EffectsRD::fsr_upscale(RID p_source_rd_texture, RID p_secondary_texture, RI
 	RD::get_singleton()->compute_list_end(compute_list);
 }
 
-void EffectsRD::taa_resolve(RID p_frame, RID p_depth, RID p_velocity, RID p_prev_velocity, RID p_history, Size2 p_resolution) {
+void EffectsRD::taa_resolve(RID p_frame, RID p_depth, RID p_velocity, RID p_prev_velocity, RID p_history, Size2 p_resolution, float blend_factor) {
 	memset(&TAA_resolve.push_constant, 0, sizeof(TAAResolvePushConstant));
 
 	TAA_resolve.push_constant.resolution_width = p_resolution.width;
 	TAA_resolve.push_constant.resolution_height = p_resolution.height;
+	TAA_resolve.push_constant.blend_factor = blend_factor;
 
 	RD::ComputeListID compute_list = RD::get_singleton()->compute_list_begin();
 	RD::get_singleton()->compute_list_bind_compute_pipeline(compute_list, TAA_resolve.pipeline);
diff --git a/servers/rendering/renderer_rd/effects_rd.h b/servers/rendering/renderer_rd/effects_rd.h
index 3bc05d7656..bb5d5149e9 100644
--- a/servers/rendering/renderer_rd/effects_rd.h
+++ b/servers/rendering/renderer_rd/effects_rd.h
@@ -94,7 +94,8 @@ private:
 	struct TAAResolvePushConstant {
 		float resolution_width;
 		float resolution_height;
-		int pad[2];
+		float blend_factor;
+		int pad;
 	};
 
 	struct TAAResolve {
@@ -668,7 +669,7 @@ public:
 	bool get_prefer_raster_effects();
 
 	void fsr_upscale(RID p_source_rd_texture, RID p_secondary_texture, RID p_destination_texture, const Size2i &p_internal_size, const Size2i &p_size, float p_fsr_upscale_sharpness);
-	void taa_resolve(RID p_frame, RID p_depth, RID p_velocity, RID p_prev_velocity, RID p_history, Size2 p_resolution);
+	void taa_resolve(RID p_frame, RID p_depth, RID p_velocity, RID p_prev_velocity, RID p_history, Size2 p_resolution, float blend_factor);
 
 	void cubemap_roughness(RID p_source_rd_texture, RID p_dest_texture, uint32_t p_face_id, uint32_t p_sample_count, float p_roughness, float p_size);
 	void cubemap_roughness_raster(RID p_source_rd_texture, RID p_dest_framebuffer, uint32_t p_face_id, uint32_t p_sample_count, float p_roughness, float p_size);
diff --git a/servers/rendering/renderer_rd/renderer_scene_render_rd.cpp b/servers/rendering/renderer_rd/renderer_scene_render_rd.cpp
index ed987578dd..ebf9bf98a6 100644
--- a/servers/rendering/renderer_rd/renderer_scene_render_rd.cpp
+++ b/servers/rendering/renderer_rd/renderer_scene_render_rd.cpp
@@ -2358,7 +2358,7 @@ void RendererSceneRenderRD::_process_taa(RID p_render_buffers, RID p_velocity_bu
 
 	RD::get_singleton()->draw_command_begin_label("TAA");
 	if (!just_allocated) {
-		storage->get_effects()->taa_resolve(rb->internal_texture, rb->depth_texture, p_velocity_buffer, rb->taa.prev_velocity, rb->taa.history, Size2(rb->width, rb->height));
+		storage->get_effects()->taa_resolve(rb->internal_texture, rb->depth_texture, p_velocity_buffer, rb->taa.prev_velocity, rb->taa.history, Size2(rb->width, rb->height), 1.0f);
 	}
 
 	copy_effects->copy_to_rect(rb->texture, rb->taa.history, Rect2(0, 0, rb->width, rb->height));
diff --git a/servers/rendering/renderer_rd/shaders/taa_resolve.glsl b/servers/rendering/renderer_rd/shaders/taa_resolve.glsl
index 197cd99f3e..5428909c73 100644
--- a/servers/rendering/renderer_rd/shaders/taa_resolve.glsl
+++ b/servers/rendering/renderer_rd/shaders/taa_resolve.glsl
@@ -53,6 +53,7 @@ layout(rgba16f, set = 5, binding = 0) uniform restrict writeonly image2D output_
 
 layout(push_constant, std430) uniform Params {
 	vec2 resolution;
+	float blend_factor;
 }
 params;
 
@@ -327,7 +328,7 @@ vec3 temporal_antialiasing(uvec2 pos_group_top_left, uvec2 pos_group, uvec2 pos_
 	color_history = clip_history_3x3(pos_group, color_history, velocity_closest);
 
 	// Compute blend factor
-	float blend_factor = RPC_16; // We want to be able to accumulate as many jitter samples as we generated, that is, 16.
+	float blend_factor = RPC_16 * params.blend_factor; // We want to be able to accumulate as many jitter samples as we generated, that is, 16.
 	{
 		// If re-projected UV is out of screen, converge to current color immediatel
 		float factor_screen = any(lessThan(uv_reprojected, vec2(0.0))) || any(greaterThan(uv_reprojected, vec2(1.0))) ? 1.0 : 0.0;

[jcostello]

I found some jittering in some cases, is that expected?

jitter.mp4

[clayjohn]

Looks great so far! The implementation is nice and clean and should provide a great base to build off of.

I am noticing the same jittering artifacts as jcostello, predominently on thin objects, but also on the edges of objects that strongly exhibit aliasing.

[mrjustaguy]

I have noticed extreme jittering artifacts on Humans (Females especially) from the side when in motion (No animations) and other objects with highly curved silhouettes also seem exhibit this behavior in motion, and movement of shadow casters at some speeds causes shadows to get horizontal lines appear at equal distances.

Aside from that, I'm loving it, it's Great at Transitioning between different Split Resolutions fairly seamlessly, and plays interestingly with MSAA (in a good way)

[Calinou]

I haven't checked the latest revision of this PR yet, but remember that the editor only redraws when needed by default. For TAA to work in still scenes¹ in the editor, it needs to request the redraw of 16 frames² for TAA to fully converge before the editor starts sleeping. This will increase CPU/GPU usage when TAA is enabled in the editor, but it's worth the quality increase. (This could be disabled when "vital redraws only" is enabled, once it's ported to 4.0.)

Edit: I tested the current revision of this PR and can confirm it doesn't force redrawing until TAA has fully converged.

Otherwise, you'll need to enable Update Continuously in the Editor Settings to ensure TAA can converge in still scenes in the editor. See also #54892.

As for jittering issues, I'm not sure how other engines tackle this. Do they decrease jittering over time when the camera doesn't move? This would make aliasing more noticeable over time though.

Footnotes

1. A "still scene" is a scene with no active AnimationPlayers, emitting particles, or shaders using TIME. ↩

2. From a quick look at the code, 16 frames appears to be the duration of the jitter sequence. ↩

[JFonS]

I spent the last days going over the implementation and double-checking that everything was working as intended. I force-pushed a new version which adds a way to disable subgroups and thus allows for shader debugging with RenderDoc.

I've been comparing our implementation with other open-source engines and also played around with some of them, and it seems like the jittering of thin bright lines is just something you have to live with. I added a bit of code from (Anki3D)[https://github.com/godlikepanos/anki-3d-engine/blob/master/AnKi/Shaders/TemporalAA.glsl] to reduce jittering, and it improves things slightly but the jittering is still noticeable.

@clayjohn Do you have a MRP or test scene for the jittering on aliasy edges you mention? I haven't really been able to find any clear examples myself.

@Calinou do you know how the editor forces continuous updates when the information panel is enabled? I was just checking so I can keep it consistent, but I can't seem to find how it's done.

Once I add the editor update code, I think this is good to review/merge. We can look for improvements and fixes later, but it's a pretty large PR and I would like to get it merged before it creates a massive conflict :)

[Calinou]

do you know how the editor forces continuous updates when the information panel is enabled? I was just checking so I can keep it consistent, but I can't seem to find how it's done.

The editor doesn't actually force constant redrawing in that situation. Instead, constant redrawing happens when View Frame Time is enabled simply because the frametime reported changes every frame since #54808 was merged. (That might be a good reason to revert this change, since it increases CPU/GPU usage a lot whenever this panel is enabled. Think of laptops' battery life 🙂)

Edit: I've tried to revert #54808 locally and constant redrawing still occurs whenever the frame time panel is enabled, regardles of the current framerate. I don't know why.

If you want to force constant redrawing, I'd look at volumetric fog temporal reprojection which does actually force constant redrawing in this case (in an explicit manner). That said, I think doing so without limiting the number of frames that will be redrawn is a bad idea as it increases CPU/GPU usage without good reason: #54893
A good implementation of this would only force redrawing for 16 frames or so.

[mrjustaguy]

Is TAA Planned for the Mobile Renderer?

[Calinou]

Is TAA Planned for the Mobile Renderer?

We discussed it on the #rendering channel on the Godot Contributors Chat a few days ago, and decided not to implement TAA in Vulkan Mobile for now. Most mobile games don't use TAA as it's fairly expensive to run compared to FXAA. That said, if there's a lot of community demand, TAA could be added to the Vulkan Mobile backend.

There are no plans to add TAA to the OpenGL renderer, given the legacy/low-end focus of that rendering backend.

[clayjohn]

Do you have a MRP or test scene for the jittering on aliasy edges you mention? I haven't really been able to find any clear examples myself.

What I used before was a sphere with a strong directional light on it from one side. With your recent pushed changes I can no longer reproduce the issue.

Also, the change you made to reduce the swimming artifacts appears to have done quite a bit! While there is still some swimming, it isn't as obvious and jarring as before. @jcostello Can you test again in your test scene that you posted earlier?

I'll start an in-depth review now!

[jcostello]

The jittering improved a bit 👍

[clayjohn]

Looks great! I appreciate how simple you kept it, it shouldn't be too much work to iterate on

Just a few comments to clean things up before it is ready to merge.

[clayjohn]

I put together a little testing project to test reprojection and I think reprojection isn't quite working perfectly yet. My test case has three objects, one is translated a little bit each frame, the other one grows and shrinks using vertex animation and the last is stationary.

For slow movements like this (with simple objects) I expect that the reprojection would be able to keep up and maintain moderate levels of anti aliasing. However, you can see that the moving objects are just as aliased and there appears to be a small amount of ghosting

TAA on

TAZA off

Test project

[Calinou]

@clayjohn The Label3D also appears to be quite affected by ghosting. I wonder if there should be a material property or render mode to disable TAA on specific materials, including the one used on Label3D and SpriteBase3D by default. Alpha-blended materials in general don't benefit much from TAA in most cases. Alpha scissor materials do benefit from TAA, but these are not the default for Label3D and SpriteBase3D.

[JFonS]

Alright, I had introduced a regression in a last-minute change. In order to save an extra buffer I used the same image as source and destination, which should work fine for local groups as they load to shared memory at the beginning, but I failed to realize that they also access neighboring group's data, which may have already been processed. This should be fixed now.

I also tweaked the disocclusion threshold and scale factors, so they are not as aggressive. I managed to track and fix an upstream bug, only to realize it had already been fixed upstream in the meantime (oh well). Plus, I applied @clayjohn's suggestions.

With all that done, there's still some minor aliasing in moving edges, but as I understand it arises from history clipping, which is essential to TAA. Most games and engines deal with it by adding a small amount of motion blur.

I can't really think of anything else to improve right now, so if the code and style look good, this should be good to merge.

[mrjustaguy]

The Issues I've had (#61319 (comment)) have been resolved in the latest branch

[clayjohn]

Looks good! It looks much better in my test scene, I also tried it out on Sponza and it looks amazing.

The last peice is to add the Licence attribution to COPYRIGHT.txt

Looks like here will be the best spot.

godot/COPYRIGHT.txt

Line 120 in 204f260

[JFonS]

So in this case, what should I list the license as? MIT or Expat? Poke @akien-mga

[akien-mga]

It should be Expat in the COPYRIGHT.txt as it follows the Debian machine readable specification and that's what they use for MIT. See #61572 as reference.

[jcostello]

@JFonS when I create a triplanar material, I get a crash on your TAA branch

[JFonS]

Force-pushed the license attribution and the fix for @jcostello's crash.

[JFonS]

Fixed the license.

[clayjohn]

Great work!

[akien-mga]

Thanks!

[MisfitVillage]

It's awesome! Sadly it makes VoxelGI flicker (I also toggle on and off FXAA to show that does not affect the flickering):

2022-06-15.20-22-53.mp4

:

[Calinou]

@MisfitVillage Please open a new issue with a minimal reproduction project attached. Also test this with SDFGI, just to make sure.

Edit: Issue opened: #62080