Vulkan: Objects close to the camera jitter a lot when far from the origin (even with float=64)

[Zylann]

Godot version

v4.0.alpha.custom_build [dae843869]

System information

Windows 10 64 bits GLES3 GeForce GTX 1060 6GB/PCIe/SSE2

Issue description

I've been working lately with a version of Godot 4 compiled with double precision (float=64), which has proven to work quite well for stuff happening CPU-side like physics.

The main problem is graphics. Everything is jittering, as if it was still working with 32-bit float transforms.
Of course a cheap workaround is origin shifting, however the point of using doubles here is so that we don't actually need to go down that burden. For graphics, it seems there is a more adapted solution to make things easier, which is to use camera-relative rendering. I think far away objects can jitter too, but because they are far, it should not be noticeable.
There are other related problems like triplanar mapping. If configured with world positions, the result is gibberish.
There are also shadow artifacts, maybe more.

Note: another topic is how to render stuff very far away. For example, if I place something at the origin and move 1000Km away, I'd like to still see it. However this issue is about stuff near the camera.

Steps to reproduce

Compile Godot with float=64, open the editor, place anything at (1000000, 1000000, 1000000), press F to move towards it, and move around it. Observe jittering.

Minimal reproduction project

TestDoubles.zip

With a version of Godot compiled with float=64, run main.tscn.
Rotate camera with middle mouse button, pan around with shift, move with WASD.
Click button on top to switch near the origin or back far from it.

• The origin of every node jitters visually, except decals
• Instances within a MultiMesh jitter independently
• When directional shadows are enabled, shadow artifacts show up randomly
• The animated cube is fine locally
• World triplanar is gibberish

[marstaik]

This is an issue caused by precision loss due to your frame of reference still being incredibly far.

It is important to understand that although the engine is computing in double precision space, you are truncating those computations back down into 32 bits of precision, and as you get further and further away and the significant digit count increases, the limited amount of precision you have in the mantissa diminishes, causing flicker. This will probably be even more exaggerated with the use of a z-buffer, which is even more finely constrained in some engines to 256-ranged values. In fact, most flickering I have seen has been due to precision issues + z-buffer depth flicker.

How can you solve it? It should be quite easy since you are now in 64 bits of precision.

Since you have compiled the engine in 64-bit precision, you actually should have more than enough precision to perform camera-based rendering like you said with introduction of very very minimal error. However, you don't even need that much. You should be simply be able to get away with camera-origin-based rendering. Simply take the inverse of the camera of the cameras translation and pre-apply it to the scene's main transformation (the top-level most) and simply remove the camera's translation in the render pipeline.

After the final object transforms are computed, you get object coordinates in double-precision form from the CPU in camera-translation space, and when truncated down to 32-bits since your reference is now closer to the objects (the camera in this case), your significant digits are reduced and you now have more precision in the mantissa.

The problem you need to solve is to reduce the amount of significant digits of the positions of objects closest to the camera, such that subsequent multiplies on the GPU can utilize as much mantissa as possible, reducing the amount of flicker near the camera.

And actually, using a full camera-space system (including rotation) would actually be worse than simple translation, because you are going from a top-level orthonormal basis built on using exact 1, 0, -1 to using an imprecise orthonormal basis (because the camera is rotated, and rotations that do not lie exactly on precise axii will introduce even greater error when inversed).

In fact, the last point is more important than it seems. For example, in blender if you export with the default FBX exporter, it performs change of basis using rotation matrices based on sin's and cosine's of rotation instead of simply flipping signs and reordering the basis vectors. This will cause havok when imported into game engines, as points in space that were once perfectly square or orthogonal are now skewed due to precision error introduced in the change of basis.

TL;DR - Apply the inverse translation of the camera to the top level matrix of the scene, and strip the translation from the camera when sent to the GPU.

The downside of this approach (and similarly, but not as much with origin shifting) is that you must recompute the final transformation matrices OF EVERY OBJECT IN THE SCENE, EVERY FRAME. Since you must perform the change of reference in the realm of higher precision, before it is sent truncated to the GPU, and your reference frame is constantly changing because the camera is moving, you cannot cache these transforms (in the classical sense). Even static objects will need new transforms every frame. This happens with origin shifting as well, except the shift and mass re-computation only occurs when the origin shifts.

You can however optimize this a little bit. Since you are double precision space, you can simply keep all of your transforms in origin space, and only apply the change of reference frame right before you send the transform to the GPU. This essentially keeps the scene transforms cached in origin 0 space, and you change its frame in double precision space right before you send it to the GPU. However, it still has the overhead of a transform multiply for every object, every frame.

Very fortunately however, since we only want to move the origin point (and not the actual basis vectors) we can cheat even further. We can simply take the final homogeneous transformation matrix sent to the GPU, and perform a single subtraction operation on the translation portion of the matrix.

[aaronfranke]

@marstaik Excellent writeup. Indeed camera-origin-based rendering is all we need, such that the only thing we need to move around is the translations of objects. The rest of the basis such as rotation doesn't need to be changed.

Do you think that you would be able to implement this feature in Godot?

[marstaik]

@aaronfranke I have not been using Godot for a while now, and do not have time myself to do this. But I think this should be relatively easy for anyone to implement.

[Gnollrunner]

As someone who hasn't used godot yet (or in fact any game engine ), Here's what I am currently doing in DirectX. I have an object hierarchy which the camera itself is actually part of. The hierarchy stores transformations both going up and down in branches of the DAG. I start at the camera which is usually at leaf and push and pop transformations as I traverse the DAG, rendering as I go. Not sure if this is what you guys are calling origin shifting, but everything starts with the camera. For things like chunks that don't need to get rotated it doesn't have to do a full matrix multiply. This is all done in double. When I send a matrix to the GPU, it converts it to float at that time. So basically I go directly to view space without going to world space first. This all works except for at very long distances, like thousands of kilometers. For that I found I had to take projection out out of the matrix and just do it as a post step on the GPU which seemed to solve the issue. Not sure if anything is relevant for godot but there you have it.

[albinaask]

if anyone would like to co-write or guide me into how to accomplish this I'd be delighted to learn. I've written a render engine and am familiar with what has to be done on an abstract level, but not too familiar with the code base below the level of the servers, nor with advanced c++.

[roalyr]

I am not sure how relevant it is, but here is my implementation of velocity-adjusted origin rebase (occurs a few times a second or less, not a continuous transform adjustment):

https://github.com/roalyr/GDTLancer/blob/master/Scripts/Ship.gd

(The code is not refactored, a proof of concept)

_physics-process has the first part, which is just about shifting environment around origin when character gets way too far away from the origin and jitter kicks in (at your desired threshold).

_integrate_forces has the second part which takes into account body linear velocity to adjust said threshold. This means that no matter how fast you go, origin rebase will happen a few times a second at most.

Naturally, with larger threshold, you will have more jitter, and my workaround was to increase camera-ship distance and camera fov, and at some point just to hide the model itself. Sure thing, with just double-precision spatial coordinates it would be much better, origin rebase threshold could be more and jitter will be less.

You can play with it by grabbing a fresh release of GDTLancer.

[roalyr]

It is also worth mentioning that not just rendering suffers from precision loss (does it even?), but object positioning and physics too, so I can't really tell if it is the right way to do it in rendering pipeline only.

[Zylann]

@roalyr did you try with doubles though? As I mention in the first post, it works ok for distances like 1000Km away (note: you might want immensely more than that but I'm not sure if Godot will support that natively as easily)

[marstaik]

It is also worth mentioning that not just rendering suffers from precision loss (does it even?), but object positioning and physics too, so I can't really tell if it is the right way to do it in rendering pipeline only.

There is already the assumption that you are using double precision on the CPU side - you would have to have an extremely, extremely large world to start getting into mantissa issues on the CPU side at that point - not that it isn't possible, but at that point you should probably be fragmenting your world into multiple origin zones anyways...

[roalyr]

I didn't try 4.0 as of yet at all, since I need to get my product done sooner than later and beg for donations...

So I am sticking with 3.x and slowly tweaking what I have figured out.

Maybe at some point I'll formalize my origin rebase approach as an asset node, or something. It is not really very straightforward and a bit convoluted (manipulates two kinds of spatial nodes, reads linear velocity, has some tweakable values, etc).

[TechnoPorg]

This will have to be implemented for GPU particles as well, since their transforms are calculated on the GPU. I believe it's done camera-independently, so I'm not sure how that might work.

[XutaxKamay]

Here is what I do personally, and it seems works well in my 3D space game, though let me know if there's an issue with it. (I'm using Godot 4.0 right now with doubles, but this can be applied to 3.4.4 as-well, just replace Node3D by Spatial)

	### To call in _physics_process for your 3D ship in open space for example ###
	move_from_all_nodes(get_tree().root, delta)
	
func move_from_all_nodes(parent, delta):
	for child in parent.get_children():
		if child == self:
			continue
		
		### If it's not a Node3D, then go deeper, might be a Node3D inside ###
		if not child.is_class("Node3D"):
			move_from_all_nodes(child, delta)
			continue
		
		### Otherwise just apply global transform, ###
		### since the transformation is not relative to parent. ###
		### All the Node3D children should move along with, ###
		### so it is not needed to move the origin for deeper in the tree, ###
		### which should save some performance ###
		child.global_transform.origin -= (abs_velocity) * delta

Please note:

• First, one of the problems that you can encounter, is that you have to handle collisions manually with move_and_collide with test_only = true, because you are not supposed to move your camera, (so your own object that you're looking into), but moving others objects. I don't know why I said this but it's total bullshit, you can just use the new origin that move_and_collide after being called gave and apply it to the others Spatial objects
• Secondly, if you have a lot of Node3D/Spatial objects that are nested deeper inside the tree, you might encounter some performance issues maybe, but I made it so it could prevent some of it by just ignoring children nodes when it can.

On my side, my camera looks very smooth now (no matter how far I am). Objects far away even though they might loose a bit of precision while moving due to floating point errors (since it modifies all origins and camera stays at position [0,0,0]), it doesn't matter, exactly because they're far away objects, we don't need that much precision for them.
What is great is that it even seems actually increases precision when objects are closer to camera/to my object I'm currently looking at.

I think it's pretty close (or it may be actually is) to what @marstaik said, don't move around what you're rendering, instead, make the camera itself "rendering" objects.

Quite important:

• By the way, if you can attach a new Node3D/Spatial to all your Node3D/Spatial objects (except the Camera and the object you want to move from), you actually don't even need to traverse the children. Just get the node and move the (global) origin. (which is what I'm basically doing now)