Add frame delta smoothing option

[lawnjelly]

Frame deltas are currently measured by querying the OS timer each frame. This is subject to random error. Frame delta smoothing instead filters the delta read from the OS by replacing it with the refresh rate delta wherever possible.

This PR also contains code to estimate the refresh rate based on the input deltas, without reading the refresh rate from the host OS.

The delta_smooth_enabled setting can also be modified at runtime through OS::, and there is also now a command line setting to override the project setting.

About

I actually originally wrote delta smoothing for Godot a couple of years ago (and shipped it in a couple of games I think), but we finally discussed it in a physics meeting a couple months ago and people were keen to try it out, so I've finally got around to it.

As well as giving nice stable deltas in vsynced games, it also takes care of hiccups due to queue stuffing (which is the method we use to restrict frame rate with vsync) - where you get a super long frame followed by a super short frame. These occur every now and then in Godot.

See #30791

Types of variation that are solved

This is the primary effect, and is particularly useful when using fixed timestep interpolation (without interpolation, these random variations can be masked, but are more likely to result in gross errors where too many or too few physics ticks are run).

Note that in some situations you can have some fluidity in the relationship between real time and game time, which can have implications for multiplayer. We could alternatively keep track of offsets and do a slow correction (maybe that could be an improvement) to keep a broad agreement between game and realtime on average.

This is very common source of jitter, where you get a slow frame followed by a fast frame, or vice versa. No frames are dropped, but because of the input timing, you see a visible glitch.

Notes

• Be aware the estimator takes a good few seconds (possibly 20 or so depending on the situation) at startup to make an estimate of refresh rate before smoothing will take effect. This will only be obvious if you compile with the debug define.
• We may come up with a better estimator in time, but this seems to do the job for an initial version.
• Just because smoothing is switched on, doesn't mean it will take effect, unless the algorithm decides it is appropriate. This usually means that high performance games will get even smoother, games that run very rough with lots of dropped frames may not benefit.
• I've also tried smoothing using a rolling average, but this method has produced the best results in my tests so far.
• It switches off if vsync is set to off in project settings, however there is the possibility of the OS overriding this, so it can cope with non-vsynced input (it does not smooth it though).
• Coping with ideal situation is very easy - where most frames are at the refresh rate. Most of the work comes from dealing with establishing an estimate of the refresh rate, and deciding when not to apply smoothing.
• This will need to be tested as widely as possible, with different monitor refresh rates, and especially variable refresh rate monitors (I'm not quite sure what will happen there yet, I may need to adapt).
• There is the possibility of reading the refresh rate from the OS, which may be preferable in the long run, although estimating it like this is a nice fallback and multiplatform.
• Once this PR is settled we can do the same in master.
• As well as giving smoother gameplay, it may also aid in capturing videos (not tested).
• It may be worth setting jitterfix to 0.0 depending on the game. Definitely with fixed timestep interpolation jitter fix should be turned off. With physics tick rate set to 60 on a 60fps refresh monitor you should now get perfect lockstep for instance, so no need for jitter fix.
• If given a completely varying frame rate, in practice the estimator will go up and down and take a long time (if at all) to settle. Then if it does settle it should not end up smoothing many frames because of the logic. This is correct behaviour, we don't want to attempt smoothing with a variable frame rate.

[lawnjelly]

It is kind of difficult to capture with video (especially as my PC is low power for running OBS studio and Godot), but here is an attempt .. without delta smoothing - stutters and hiccups and irregular deltas:
https://user-images.githubusercontent.com/21999379/116851262-fb7dfe00-abe9-11eb-965e-d9028089f894.mp4

And with (there are still a couple of stutters, but caused by recording):
https://user-images.githubusercontent.com/21999379/116851365-2405f800-abea-11eb-94fa-a859470c4341.mp4

[Calinou]

Here's some observations from testing.

• Project: https://github.com/Calinou/escape-space
  • Changes made from the master branch: delta smoothing enabled, V-Sync enabled and the jitter fix disabled.
  • This project uses the smoothing add-on. Pixel snap is disabled. The player's Camera2D is set to run with Idle update mode.
• OS: Linux
• Display server: X11
• Graphics driver: NVIDIA proprietary driver
• Monitors: 3 × 144 Hz monitors, all with the same resolution

---

• The default initial estimate and actual estimate with all monitors set to 144 Hz is correct (144 FPS).
• If all monitors are set to 60 Hz and the game is restarted, the estimates are still correct. I get occasional hitches while moving though (the player appears to be skipping around).
  Log:

initial estimate of refresh rate: 60
estimated fps 102
estimated fps 123
estimated fps 134
estimated fps 139
estimated fps 140
estimated fps 142
estimated fps 143
estimated fps 144
estimated fps 145
estimate complete. vsync_delta 6896, threshold 8275
hits at estimated : 6, above : 1( 0 ), below : 4 (1 )
hits at estimated : 12, above : 1( 0 ), below : 4 (1 )

• If the main monitor is set to 60 Hz but other monitors remain at 144 Hz, the game is estimated to be V-Syncing at 144 FPS (tested after restarting the game while on the main monitor).

• If I set the left monitor to 60 Hz, the center (main) monitor to 120 Hz and the right monitor to 100 Hz, the game estimates the refresh rate at 60 Hz but the actual FPS is around 100.
  Log:

initial estimate of refresh rate: 60
estimated fps 102
estimated fps 101

• With the setting changes mentioned above applied, I don't get occasional stutters anymore. I only tested a few minutes of gameplay with each Godot binary (stock vs this branch), though.
• The player still jitters when moving, regardless of the jitter fix value. This may be an issue with Camera2D though. Balls also appear to jitter when they move in the same direction as the player.
  • The most jitter-free way of player movement is, ironically, with V-Sync disabled. (With V-Sync disabled, I get 300+ FPS on the project here.)

I can test this on Windows too if you want, since I have a dual boot setup.

[lawnjelly]

That's quite a thorough test, as having 3 monitors has got to be a worst case scenario! 😄 (along with variable refresh rate which we must test)

There is a danger of confusing two separate issues though - delta smoothing, and our multi monitor support. It is easier to examine one at a time in isolation first before mixing the two.

The estimation is based entirely on the input deltas that are coming in. Worth noting is the PR is an attempt to give a slight (and often subtle) improvement over the default behaviour, but it cannot and does not attempt to correct for all hitches. I've tried to take a conservative approach and err on the side of caution, because over tuning for a particular situation can potentially cause problems when running with an inconsistent frame rate. We can of course attempt to improve this over time as we get more test cases (and the estimator).

Generally the current strategy is to turn the smoothing off automatically if the delta is all over the place. I'm particularly interested in any scenarios where delta smoothing on may lead to a worse experience than with it off.

Multi monitor setup

To summarise, multi monitor with varying refresh rates may not be something that this PR can help with. The best you can hope for is that, by chance, the vsync rates you get are from the monitor you are using godot on, in which case it will help. The only thing that concerns the PR directly is that it should not negatively impact multi monitor with varying refresh rates. And it should not, it should either converge on a regular vsync rate (and give roughly similar behaviour), or not converge, and take no effect.

I actually have no idea how godot currently deals with vsync on multi monitors with varying refresh rate. It is quite possible that we don't have a sensible strategy for multi monitors at the moment (which is kind of related, but not specific to the PR).

Do we know if it currently vsyncs on the monitor that Godot is displaying on (out of the 3)? You don't mention in the results which monitor godot is displaying on. Or on the fastest / slowest? Or does a vsync occur for all of the 3 in a stuttering fashion? This may be very OS / driver dependent. It would be nice to get some logs of deltas on your machine with these setups (although beware that printing the logs will affect things, so storing up e.g. 100 deltas then printing at once may work better, or printing some summary statistics like the variance etc). The fact the estimator is converging at all does suggest it might be vsyncing at just one of these refresh rates.

(Actually the first appears to be converging on 145fps, rather than 144, it may be your refresh rates are closer to 145 in practice. The second isn't converging from the log you posted, so maybe there aren't regular vsyncs. It only converges when it says estimate complete, and only remains 'sticky' once the hits at estimated reaches 20. So there shouldn't be any smoothing in the second case.)

If you are getting the best results with vsync disabled when you have the combination of refresh rates (I'm assuming regardless of whether you have delta smoothing on), then that suggests possible aliasing as Godot vsyncs on a particular refresh rate, and by disabling, you are increasing the probability that by chance you will get a frame that matches up to the display time.

Either way I don't think delta smoothing can always help in this scenario, unless the estimator happens to be getting deltas for the display that godot is appearing on. If the deltas are all over the place, I'd be surprised if it converged to a solution, and if it does, it is unlikely to activate many smoothed frames. There isn't much feedback on that front, but I could add some more info to the debug mode.

I'll try the game on my machine and see the results. I seem to remember it ran very smoothly for me last time I tried it. 👍

Trying it all on windows will I'm sure be interesting too.

[lawnjelly]

Ok trying the latest Escape Space game on mine:

Firstly I needed to turn on vsync otherwise I got bad tearing. Also I set jitter fix to 0.0, as setting to 0.0 is much better with fixed timestep interpolation, otherwise the jitter fix will muck up the results (jitter fix pushes and pulls timesteps around so they are irregular). I saw you did that on your test, maybe it just needs the repository updating.

I also ran at fullscreen because this tends to give better results, maybe OS gives a higher priority. I'm running Linux Mint, and monitor is 60fps. For me the game and movement was very smooth both with and without delta smoothing. However it ran marginally better with delta smoothing.

Judders - you are correct, running normally every now and then there is some juddering perhaps aliasing between the camera and the player physics position. With delta_smoothing, I couldn't get it to do the judders, it was fully smooth. I'm not exactly sure of the cause of the judders without looking at the details of the game. And it could be that the delta smoothing helped especially because the physics tick rate and the display refresh rate were 60 hz on my system. Yes I can confirm, if I set the physics tick rate to 59, the nasty periodic judders are back. You are probably correct that this is due to a mismatch between your camera movement and the player. I've a feeling I've solved this in the past by moving the relative position of the camera and the player in the scene tree as it can be an order of processing thing with the interpolation node, I keep meaning to fix that with process priorities.

So yes it's quite important to have a good reference project to try (a ground truth), even if it is a very simple project. I used both the smoothing node demo (which ticks physics at something like 8 tps) and the truck town demo (which has no fixed timestep interpolation but you can vary the physics rate).

Getting a little off topic but:
Looking at your project I think the problem is that you have made Camera2D and the smoothing node children of the object (Paddle rigid body) you are trying to follow. I always recommend against this although people have been doing this. Although it appears logically correct it has the potential to create feedback loops. I'll see if I can fix it for your project.

Yes this seemed to be the cause of your judder. If you put the physics tick rate to e.g. 1 out from your display refresh rate it will accentuate the effect. The solution is to make sure the smoothing node is not a child or grandchild of the node you are following. I think I have prevented this in the 3d version but I reverted the change in the 2d version because people wanted to use it like this. But perhaps I should reintroduce the limitation, I'm not sure:

With this modification your game runs absolutely perfect at 59 ticks per second on 60 fps refresh rate monitor (once the estimator has converged), with delta smoothing. Without delta smoothing you still get occasional hitches.

Managed to get this in a capture by decreasing the window size so OBS studio can keep up (!) 😄
With delta smoothing
https://user-images.githubusercontent.com/21999379/117495165-2f03b400-af6d-11eb-9443-255dcaa473e2.mp4

Without delta smoothing
Extra judder is very visible.
https://user-images.githubusercontent.com/21999379/117495213-3f1b9380-af6d-11eb-919b-f7d43c1fdaf3.mp4

[Zireael07]

I think I have prevented this in the 3d version but I reverted the change in the 2d version because people wanted to use it like this. But perhaps I should reintroduce the limitation, I'm not sure:

Show a yellow warning triangle in this case, maybe?

[lawnjelly]

Show a yellow warning triangle in this case, maybe?

Yes, sounds a good idea, I should update the addon with this warning and maybe the process_priority thing (I didn't realise you could change the process_priority when I originally wrote it, but this avoids having to order nodes in the scene tree to such an extent). Users are mostly shielded from order of processing of nodes, but for things like jitter it can be crucial.

[lawnjelly]

On a very related note: given the huge improvement with hitches seen in the videos in #48390 (comment), it got me wondering ..

Delta smoothing should improve things, but why is it improving it that much?

I think I may have an answer. I had long suspected (but never had a good test case) that we are measuring the frame delta in the wrong place in the game loop.

The time for the 'frame start' is currently measured at the beginning of main::iteration. However it had previously occurred to me that this might not be the best time to measure the frame time when using vsync. To me, the best time is exactly when OpenGL returns after blocking. This could either be after some OpenGL command, or swapping the buffers.

By doing a load of other variable work in between that point and then measuring the time (possibly returning control to the OS) we may be adding a huge amount of unncessary random variance to the timing measurements.

Anyway for a trial I tried moving the 'frame start' with the following modification:

uint64_t g_global_frame_ticks = 0;

bool Main::iteration() {
	uint64_t ticks = g_global_frame_ticks;
	
	if (!ticks)
		ticks = OS::get_singleton()->get_ticks_usec();
	
// PHYSICS
// IDLE
// DRAW
	g_global_frame_ticks = OS::get_singleton()->get_ticks_usec();

// SCRIPTSERVER
// AUDIO
// SCRIPT DEBUGGER
// etc
}

i.e. I moved the sync point for taking the time to immediately after the draw.

Result: it seemed to produce hugely reduced stuttering, even with delta smoothing off, in this test case with Calinou's game.

I'll do a bit more investigation in this area but I suspect there will be a good case for moving this time measurement. But it will be for another PR.

[lawnjelly]

Very improved version now. It deals with drift from wall clock time (drift is usually less than a frame), uses a simplified algorithm.

There are now three protections against activation when vsync is not on:

• Tests the engine setting
• Estimator is unlikely to converge
• Rough fps measurement at runtime. If it is significantly higher or lower than the estimated FPS, smoothing is deactivated.

[lawnjelly]

Good feedback from #33969 (comment) .

Based on the feedback there, it may be reasonable to disable delta_smoothing when vsync_with_compositor is set on (or vice versa, to disable vsync_with_compositor), however it is too early to make this call, I think we should wait for some results in a beta before changing this.

[akien-mga]

Thanks!

[oeleo1]

@lawnjelly I have a question: what are the chances of this thing desynchronizing during gameplay after the initial sync, and if they are greater than zero, under what circumstances would that happen?