Add stage with testing project in CI to find regressions earlier

[qarmin]

Describe the project you are working on:
I working in Godot 3.2 with 3d project, but due a lot of new features I want to upgrade to Godot 4.0

Describe the problem or limitation you are having in your project:
For now my testing branch with Godot 4.0 is very unstable, because every few days or weeks some parts of engine just stop working or working bad.
I know that is too early to port projects to unstable version of engine, but also some regression I spotted in Godot 3.2.x(smaller of course)

Describe the feature / enhancement and how it helps to overcome the problem or limitation:
Automatic running example project with help of address, leak and undefined sanitizers should help to catch more bugs and crashes before merging PR.

This will help to find:

• Memory leaks - CI will fail
• Crashes - CI will fail
• Invalid memory read/write - CI will fail
• Undefined code behaviors - they will be printed, but CI may pass
• Errors printed to console - CI may pass

This will NOT help to find:

• Invalid logic - like invalid size of object or colors of shader
• Editor bugs which happens after some user changes

Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams:
I implemented this here - godotengine/godot#40994, but still needs testing and new test project.

In this PR:

• Godot and release build are compiled with debug symbols
• Project https://github.com/qarmin/RegressionTestProject/tree/3.2 is downloaded
• Godot editor opens project, import it and close
• Godot tests project by running it
• Project is exported
• Project is running as external app

This for now finds 3 bugs which blocks this PR, but in future may find much more.
It is sadly unavailable in Godot 4.0 since it use Vulkan

My test project is too basic and needs to be changed to something more complex, but for now it is enough good to simply test most basics regressions.

New project should:

• give easy to reproduction results(no randi or similar function)
• be small(for me max is few MB - current assets size, not all git repository)
• test big amount of nodes and resources(2D, 3D, Shaders, GDScript, Physics etc...)

If this enhancement will not be used often, can it be worked around with a few lines of script?:
This will be used on every PR, and alternative is to manual testing engine as it is today.

Is there a reason why this should be core and not an add-on in the asset library?:
This need be added to build system not core or asset library

[qarmin]

Godot recently get ability to use doctest which simplify use of unit tests.
Now old tests are ported to new system so with time, tests should be more complex and catch more bugs at compile time.

[Xrayez]

I think having ability to test projects on CI would still be quite beneficial even with doctest already integrated, mostly because projects better reflect real-life usage of the engine, so this would certainly help to identify regressions more easily than just running raw C++ unit tests, and could also provide some solutions to #168 as well.

Improving import process would also be useful as described in #1362 for this to work with reproducible results, as you said.

Mainly talking about issues like godotengine/godot#42305 and godotengine/godot#42339, these regressions were detected "promptly" because I have a C++ module which does run some tests using GUT testing framework written in GDScript, even using mostly the same GitHub Actions, so something similar could be implemented in Godot to prevent this even earlier in the development stages. I guess the test projects doesn't even have to be too sophisticated for this to yield good results.

This is especially important for stable branches, in fact I have another proposal which suggests backporting doctest to the 3.2 in the interest of stability: #1533.

[qarmin]

Now PR is merged and 3.2 branch is tested with this project - https://github.com/qarmin/RegressionTestProject/tree/3.2 and even found some leaks/crashes.

GUT leaks a lot of memory, so if it was used in a CI, it would have to be either repaired or Leak sanitizer would have to be turned off (which I would prefer to avoid).

If anyone wants to add a unit test to Godot(in GDScript), they should create PR in this repository (branch 3.2).

I don't close this issue because I still don't know how to run Godot 4.0 inside CI, but I will try to do it with LavaPipe.

[jordo]

@Xrayez I'm just gunna say i'm on board.

godotengine/godot#43941 way too much time to find :(

[qarmin]

Looks that soon Godot 4.0 will have GLES 2 or 3 backed which will allow to easily test projects in CI like in 3.2 branch, and this will greatly improve engine stability.
https://twitter.com/lawnjelly/status/1332713710230663170

@lawnjelly Do you have any preliminary calculations when the backed one would be useful?

[lawnjelly]

I don't actually know the extent it has to be usable for CI.. In terms of plans I'd like to get a rough and ready 2d GLES2 into master in the not too distant future. A lot is working already in a few days, but it's difficult to know how long it will take to get stable and refactored (we are still getting the multiple windows working for instance, as this has been uncharted territory for us, and the work so far has been flying by the seat of your pants type stuff, it will need tidying up).

At a very rough guess maybe a couple weeks. It depends on opinions of how 'alpha' it should be to first merge. I'm of the view it would be useful to get a rough version in use so others who depend on this can at least get started on their work. Something where most of the 2D UI is working and you can make simple 2D games. I also can't entirely predict when I'll have to be away from home so this can hold things up although soon clayjohn will be able to do most stuff too.

Then we can get started on GLES3, which shares a lot of code with GLES2, so I anticipate getting the 2d for GLES3 working should be quicker, and then we can look at 3d, the details of that especially shader modifications for 4.0 will probably be more clayjohn's area.

Note there are some differences in Godot 4.0 that will take longer to get a meaningful new version for - for instance the 2d lighting has changed I believe.

Also there will need to be OpenGL context managers written for each platform. I've started with x11, @clayjohn is currently doing windows. It might not support all the platforms at first. I can probably do android but it may need a mac guy to do mac and iOS for instance.

[qarmin]

Finally with help of SwiftShader(allows to run Vulkan app on CPU instead GPU) I created PR with adding CI to Godot - godotengine/godot#47414.

Still I'm trying to find some good and short scenes which I could add to test project.
For now there are 4 types of scenes in https://github.com/qarmin/RegressionTestProject:

• Normal scenes - Physics3D.tscn just display some simple examples - this require from user look at the graphic to see difference/regressions between versions
• Fuzzer scene - FunctionExecutor.tscn - Test with always same input each method in every class
• Adding and deleting nodes - Nodes.tscn - Adds all nodes to scene and after some time removes them and create new ones
• Reparenting/moving/deleting nodes - ReparentingDeleting.tscn - Randomly move, reparents, delete or adds node to scene

[qarmin]

Both 3.x branch and 4.0 have test project in CI, so issue can be closed.