Flaky tests around Input simulation inside Scene Runner

[placer14]

I'm making some simple integration tests where I'm exercising the inputs, but they fail inconsistently and for no obvious reason. My inputs tested are just the directional movement inputs and I know they work when I run the test scene manually. My test is included below.

• I'm using godot 4.3 w gdunit 4.3.4.
• actions_fixture.tscn only contains a Player (CharBody2D) and an Enemy (CharBody2D) scene and nothing else. There are no scripts in the scene.
• When I run these tests individually, they work. When the suite is run altogether, the first one (right) passes, the other three directional tests (left, up, down) fail. And attack works. Maybe the input handling isn't correct?
• I've tried various forms of await between inputs. Trying await_millis, await_idle_frame and runner.await_input_processed without any change in outcome.

Feedback on my test pattern is also welcome.

class_name TestPlayerActions
extends GdUnitTestSuite

var _player_class = "PlayerV2"
var _enemy_class = "Enemy"


func test_player__input_movement__right() -> void:
	var _runner = scene_runner("res://tests/integration/player/actions_fixture.tscn")
	var _player = _runner.invoke("find_child", _player_class)
	var original_position = _player.position
	_runner.simulate_action_press(Constants.INPUT_RIGHT)
	await await_idle_frame()
	_runner.simulate_action_release(Constants.INPUT_RIGHT)
	
	assert_float(original_position.x).is_less(_player.position.x)
	assert_float(original_position.y).is_equal(_player.position.y)


func test_player__input_movement__left() -> void:
	var _runner = scene_runner("res://tests/integration/player/actions_fixture.tscn")
	var _player = _runner.invoke("find_child", _player_class)
	var original_position = _player.position
	_runner.simulate_action_press(Constants.INPUT_LEFT)
	await await_idle_frame()
	_runner.simulate_action_release(Constants.INPUT_LEFT)
	
	assert_float(original_position.x).is_greater(_player.position.x)
	assert_float(original_position.y).is_equal(_player.position.y)


func test_player__input_movement__up() -> void:
	var _runner = scene_runner("res://tests/integration/player/actions_fixture.tscn")
	var _player = _runner.invoke("find_child", _player_class)
	var original_position = _player.position
	_runner.simulate_action_press(Constants.INPUT_UP)
	await await_idle_frame()
	_runner.simulate_action_release(Constants.INPUT_UP)
	
	assert_float(original_position.x).is_equal(_player.position.x)
	assert_float(original_position.y).is_greater(_player.position.y)


func test_player__input_movement__down() -> void:
	var _runner = scene_runner("res://tests/integration/player/actions_fixture.tscn")
	var _player = _runner.invoke("find_child", _player_class)
	var original_position = _player.position
	_runner.simulate_action_press(Constants.INPUT_DOWN)
	await await_idle_frame()
	_runner.simulate_action_release(Constants.INPUT_DOWN)
	
	assert_float(original_position.x).is_equal(_player.position.x)
	assert_float(original_position.y).is_less(_player.position.y)


func test_player__input_attack() -> void:
	var _runner = scene_runner("res://tests/integration/player/actions_fixture.tscn")
	var _player = _runner.invoke("find_child", _player_class)
	_runner.simulate_action_pressed(Constants.INPUT_UP)
	await await_idle_frame()
	_runner.simulate_action_pressed(Constants.INPUT_ATTACK)
	await await_idle_frame()

	assert_str(_player.current_state()).is_equal(PlayerV2.STATE_ATTACKING)

Note these worked pretty consistently when I was working on them during my previous dev session. Today, I started the editor and found gdunit had an update (was on 4.3.4 originally). When I updated to 4.4, all of my Input-based tests started failing. I rolled the add-on back to 4.3.4 and now I'm in this situation. I'm not sure the update had anything related to this, but though I'd share the details.

[placer14]

This appears to be similar to #379 but using gdscript instead of C#.

[MikeSchulze]

Hi @placer14 that sounds bad.

From the internal test coverage, all tests successfully.

From the test, it misses input synchronization after the action release.
Check the documentation

e.g.

func test_player__input_movement__right() -> void:
	var _runner = scene_runner("res://tests/integration/player/actions_fixture.tscn")
	var _player = _runner.invoke("find_child", _player_class)
	var original_position = _player.position

	_runner.simulate_action_press(Constants.INPUT_RIGHT)
	await _runner.await_input_processed()
	_runner.simulate_action_release(Constants.INPUT_RIGHT)
        await _runner.await_input_processed()
	
	assert_float(original_position.x).is_less(_player.position.x)
	assert_float(original_position.y).is_equal(_player.position.y)

When you only want to test an action is pressed, you can simplify your test by:
And the scene runner provides an find_child function.

## Searches for the specified node with the name in the current scene and returns it, otherwise null.[br]
## [member name] : the name of the node to find[br]
## [member recursive] : enables/disables seraching recursive[br]
## [member return] : the node if find otherwise null
func find_child(name: String, recursive: bool = true, owned: bool = false) -> Node:

func test_player__input_movement__right() -> void:
	var _runner = scene_runner("res://tests/integration/player/actions_fixture.tscn")
	var _player = _runner.find_child(_player_class)
	var original_position = _player.position

	_runner.simulate_action_pressed(Constants.INPUT_RIGHT)
	await _runner.await_input_processed()
	
	assert_float(original_position.x).is_less(_player.position.x)
	assert_float(original_position.y).is_equal(_player.position.y)

If this not helps, please create a bug issue and append an example project to reproduce.

[placer14]

Hey @MikeSchulze. I've invited you to a private repo with the failing tests. (I'm sorry I don't want to share the example publicly. If this isn't going to work, I can try to reduce to a minimal example... but that's effort I would rather not invest right now.) Let me know if you have any ideas of what I could be doing wrong.

Additionally, I also intermittently get this error. Not sure if you've seen this, but maybe you might know why I'm hitting this, too. It occurs when I run the full test suite (Run overall tests) and this case occurred inside the test_player__enemy_touch__damange_absorbed_by_kenergy test.

Left operand of 'is' is a previously freed instance.

Unsure how to copy/paste the stacktrace, but here's a picture:

[MikeSchulze]

@placer14 I will take a look

[MikeSchulze]

The tests fixed now, see https://github.com/placer14/ktesting/pull/1
It took me a while to understand how the input processing works in your scenes.
There is a mixture of input processing in the default render and physics frame ;)

[placer14]

For the benefit of others, my flaky tests were due to my input handling. I had a singleton checking Input.get_vector twice... inside _process and in _process_physics. Unfortunately, I believe the input was being flushed before either of those hooks were called. As such, those inputs were never seen or acted upon.

Here's an overview of the changes applied. Before:

func test_player__input_movement__right() -> void:
	var _runner = scene_runner("res://tests/integration/player/actions_fixture.tscn")
	var _player = _runner.find_child(_player_class)
	var original_position = _player.position

	_runner.simulate_action_pressed(Constants.INPUT_RIGHT)
	await _runner.await_input_processed()
	
	assert_float(original_position.x).is_less(_player.position.x)
	assert_float(original_position.y).is_equal(_player.position.y)

After:

func test_player__input_movement__right() -> void:
	var _runner = scene_runner("res://tests/integration/player/actions_fixture.tscn")
	var _player = _runner.find_child(_player_class)
	var original_position = _player.position

	# `press` and not `pressed` means the input remains "on"
	_runner.simulate_action_press(Constants.INPUT_RIGHT)
	# flush the input
	await _runner.await_input_processed()
	# await a physics frame to update my player with the input engaged
	await Engine.get_main_loop().physics_frame
	
	# these now pass
	assert_float(original_position.x).is_less(_player.position.x)
	assert_float(original_position.y).is_equal(_player.position.y)

[placer14]

What I don't understand @MikeSchulze is that await_input_processed seems to also await physics_frame and wonder why the additional one outside the await_input_processed in the unit tests is needed for all of them to pass. Could you explain?

[MikeSchulze]

What I don't understand @MikeSchulze is that await_input_processed seems to also await physics_frame and wonder why the additional one outside the await_input_processed in the unit tests is needed for all of them to pass. Could you explain?

This is need because of your input handling workflow.
The Input.get_vector is processes in the physics_frame first and in the next physics_frame the before caught vector is processed.

I found this out by adding debug statements to print out the order your inputs are processed.

Do it by add a simple for loop after the _runner.simulate_action_press(Constants.INPUT_RIGHT) and do a wait physics_frame and process_frame with a frame counter to print.
Then you will see when the input is caught and when it is processes in your walking class.