Add a project upgrade tool for Godot 4.0

[BenjaminNavarro]

Describe the project you are working on:
I mostly make games on the side for fun using Godot 3.x but I follow its development quite closely.

Describe the problem or limitation you are having in your project:
When Godot 4.0 is released I would like to easily be able to switch to the new version with my current 3.x projects.

But being a major release, some parts of the API will change, as outlined in this issue. This can make the transition phase a bit tedious if it has to be done manually.

Describe how this feature / enhancement will help you overcome this problem or limitation:
Since the mapping between the old names and the new ones is known the project imported can probably be modified to upgrade 3.x projects to be 4.0 compatible automatically. If some parts of the project cannot be upgraded automatically (e.g. removed feature) they should be clearly listed along with some guidelines on how to deal with these issues.

Show a mock up screenshots/video or a flow diagram explaining how your proposal will work:
It's a bit early for this, but I can make one once I know if it's theoretically possible to do it and what would be the best solution for it.

Describe implementation detail for your proposal (in code), if possible:
Again, too early

If this enhancement will not be used often, can it be worked around with a few lines of script?:
It probably can't.

Is there a reason why this should be core and not an add-on in the asset library?:
So that current users can migrate to 4.0 as smoothly as possible. If it's separated from the editor, most people probably won't know about it and struggle to upgrade their projects.

[Boivin92]

I love this idea but I think it should be a separate tool, not integrated to the editor. The main reason being that once 4.0 is a few months old, this feature would instantly be deprecated.

We could still make the tool easy to find on the website, maybe a subsection of the "Download" page

[BenjaminNavarro]

Well, I would say that it mainly depends on the impact of the feature on the editor. Meaning that if it's small enough and doesn't bother anyone we might just leave it in the editor/project manager. And that's what most software do, including Unity (don't know much about other game engines).

That way, even if people start with 4.0 but want to import some 3.x demo project found online, it can be done seamlessly.

But I'm not in charge of the engine development so it will be what is decided to be.

Also, I'm willing to work on this when the 4.0 branch become a thing.

[BenjaminNavarro]

I just realized that something similar was already (briefly) discussed by @reduz (godotengine/godot#30736, godotengine/godot#30736 (comment)).

Considering this, do we need to keep this proposal open ?

[golddotasksquestions]

I think it can be closed once the "internal compatibility system" is added, no?

[KoBeWi]

Considering this, do we need to keep this proposal open ?

A formal proposal is always good to keep, I think.

As for this system, I thought it could be included in the breaking versions, but removed in the next version. So e.g. to upgrade from 3.2 to 4.0.1, you'd first need to import your project in 4.0. It requires some manual work, but is still less tedious than doing it yourself and we avoid uneccessary bloat.

btw, bringing up Unity here is not very good example. Upgrading from x.0.1 to x.0.1a requires 2GB download and 15 minutes of importing. In Godot you can juggle between versions pretty seamlessly and the lightweightness is a good selling point we'd rather not loose.

[qarmin]

For anyone who want to try implement this, I prepared test project in Godot 3.4 with almost all available functions which for now after opening in Godot 4 shows ~1400 errors. It would be great if the number of errors would decrease to ~100-200 using this tool.

GDScript3.4.zip

[Calinou]

@aaronfranke wrote a shell script to convert Godot 3.x projects to Godot 4.0: https://gist.github.com/aaronfranke/79b424226475d277d0035b7835b09c5f

[qarmin]

Is there any preconception how it should look like(I haven't heard anything about it lately and it's quite a serious topic)?
Will it be a separate script(which I doubt) or an additional feature in the editor?
Will it convert only .gd files or also other for example .tscn?
Will it be possible to choose individual files for conversion or will I have to convert whole project like in 2.1?

[aaronfranke]

@qarmin For the Bash script I wrote, it operates on all types of files (scenes, GDScript, C#, project.godot, .import, etc), and it operates on the entire repository at once.

[qarmin]

I'm curious what is the better option to upgrade project:

• When opening project from Godot 3 in Godot 4, prompt with question about upgrading should become visible(should be possible to not allow to broke project)
  or maybe
• Command line argument like godot4 --upgradeTo4 . should be added

This is how it upgrade tool works in 2.1 branch
https://github.com/godotengine/godot/blob/2.1/editor/editor_export_godot3.cpp

[akien-mga]

For the reference, here's another conversion script developed by @lyuma and @fire: https://github.com/V-Sekai/gd3to4

[lyuma]

I already submitted most of what I had in my script to be integrated into qarmin's pr godotengine/godot#51950 and so far qarmin has done really good work implementing most of the rules I had in my script.

A couple properties are hard to integrate because they require type information to know (such as toplevel -> top_level) if they are the builtin one or a user-defined function. Also pause_mode -> process_mode is especially confusing because the enum changed, so 2 needs to become 3 or whatever.

Also still missing some of the EditorPlugin type apis and importers, like _edit, _handles, _get_extensions etc.

One of the other things missing might FuncRef.new -> Callable : because Callable is a builtin type, you can't use .new but instead have to just create it with arguments.

[Cerno-b]

As discussed here (godotengine/godot-docs#4960), would it make sense to integrate documentation into the converter? Bear in mind that not only the projects need to be converted from 3 to 4, the developers need to as well.

If we could have a string lookup generator closely coupled to the code conversion function that map each conversion to human readable format, we could generate a reST code out of the lookup table that we could drop into the Godot docs, e.g. in the form of a table "old code" --> "new code".

I think this would be a very convenient way to get Godot 3 devs up to speed with changed syntax.

[akien-mga]

An initial version of @qarmin's upgrade tool has been merged and is available in 4.0 alpha 10: godotengine/godot#51950

We can keep this proposal open for now to keep track of further improvements to make to this tool, especially sourced from the other existing conversion tools which may handle other stuff.

[Zireael07]

First order of business, I couldn't see shader hints in that, and they're bound to break people's shaders.
PR that renamed them is godotengine/godot#60803 so way before alpha10

[KoBeWi]

Some random conversions missing in the official converter that you can find in mine:

• conversion of some properties of specific Controls, e.g. Label alignment or TextureRect stretch mode
• theme overrides
• animation rotation_degrees (it should be changed to rotation and converted to radians)
• better conversion of setters (right now the converter creates stumps, but actually var test setget set_test can become var test: set = set_test)
• particle randomness became min/max values. It's possible to convert it based on the old formulas

I'm actually surprised that the official converter already handles built-in scripts 🤔

Other things to consider:

• convert SceneTreeTween to Tween. The only API difference is class name (relevant if someone uses explicit type) and callback/method Tweeners, which use Callable in 4.0
• conversion of connect() to the new syntax? other.connect("signal", target, "method", [binds]) becomes other.signal.connect(target.method).bind(binds). This is handled in my converter, but the official one handles connections in a different way (which works), so it's ok too

[FeralBytes]

Seems that Threads has changed quite a bit and now uses a Callable.bind, as such my threading library is broke on conversion. I am working on figuring out how the change works, then I will report back.
For threading I converted:

start_success = new_thread.start(self, "__threaded_background_loader", 
    [resource_path, thread_num]
)

Into

var the_callable = Callable(self, "__threaded_background_loader").bind([resource_path, thread_num])
start_success = new_thread.start(the_callable)

Also seems that the following has changed/moved and still needs to be accounted for in the conversion:

• Built-in Scripts that are tool scripts do not get the keyword "tool" converted to "@tool"
• Engine.editor_hint to Engine.is_editor_hint()
• OS.get_screen_size() to DisplayServer.screen_get_size()
• OS.get_real_window_size() to DisplayServer.window_get_real_size()
• OS.get_window_size() to DisplayServer.window_get_size()
• String.plus_file(path) becomes String.path_join(path) this changed during the beta see here.
• MainLoop.NOTIFICATION_WM_QUIT_REQUEST to NOTIFICATION_WM_CLOSE_REQUEST
• MainLoop.NOTIFICATION_ to NOTIFICATION_
• Thread.is_active() is no longer used should be converted to Thread.is_alive()
• File.new() to the new FileAccess API per Link
• Directory.new() conversion to DirAccess API, would at least be nice if the tool pointed out these issues even if it can not handle them.
• Tween.new() is invalid and can't be used for tweening values. So these should be identified by the converter if they can not be handled. Though it should convert them too something like create_tween().
• minimum_size on Container based nodes does not get converted to custom_minimum_size.
• For some reason random "export" keyword conversions went bad when it comes to type declaration, the "export_enum" is likely a multi-line issue, although maybe not since it did not recommend the fact that export had parameters which is standard for an ENUM based export. For instance:

export(bool) var run_specified_test: bool = false
export(float) var artificial_delay_between_states: float = 0
export(int, "not_set", "debug", "info", "warning", "error", "critical", 
    "testing") var debug_level: int = 3

was converted to

@export var run_specified_test: bool: bool = false
@export var artificial_delay_between_states: float: float = 0
@export(int, "not_set", "debug", "info", "warning", "error", "critical", 
    "testing") var debug_level: int = 3

but should be

@export var run_specified_test: bool= false
@export var artificial_delay_between_states: float = 0
@export_enum("not_set", "debug", "info", "warning", "error", "critical", 
    "testing") var debug_level: int = 3

• Coroutines are handled poorly by the converter. For instance [GDScriptFunctionState](GDScriptFunctionState](https://github.com/godotengine/godot/blob/master/editor/project_converter_3_to_4.cpp#L1392):

var potential_function_state = custom_event_overrides.on_preloading_start()
if potential_function_state is GDScriptFunctionState: # Still working.
    potential_function_state = yield(potential_function_state, "completed")

needs to be converted too:

await custom_event_overrides.on_preloading_start()

But instead comes out as:

var potential_function_state = custom_event_overrides.on_preloading_start()
if potential_function_state is GDScriptFunctionState: # Still working.
    potential_function_state = await potential_function_state.completed

The most obvious issue here is that GDScriptFunctionState should be removed, so if the converter could at least point out each of these issues it will help the conversion go smoother. I will say that typing one line instead of the mandatory 3 lines before is much nicer.

• Yielding for Idle_Frame was in a ton of examples so the converter should be able to handle this, but it does not. Instead:

yield(get_tree(), "idle_frame")

Becomes

await get_tree().idle_frame

But should be

await get_tree().process_frame

Also the new documentation still has an old example here

• OS.get_system_time_secs() to Time.get_time_dict_from_system()["second"]
• WebSocketServer is still listed in documentation but referencing it causes the identifier to not be found.
• get_tree().get_network_peer() gets converted to get_tree().get_multiplayer_peer() but should be get_tree().get_multiplayer().get_multiplayer_peer()
• get_tree().is_server() no longer exists should be get_tree().get_multiplayer().is_server()

I also just found that I should run this tool first: https://github.com/Scony/godot-gdscript-toolkit
So I will do that tomorrow and see what all is fixed as a result

A Warning to any using ResourceLoader, it has changed dramatically. Fixing the issues will take time for sure, but it is actually a smarter system.

A majority of the conversion was done on Godot 4.0-beta4. I think the converter would benefit if it ensured everything was a single line, perhaps merging some code from GDScript Toolkit to make that happen.
One last note, I really appreciate all of the effort that has been put into this conversion tool. I am just pointing out where there is more work to be done.

Edit/Update (27JAN):
I found that PacketPeerUDP.listen() is now PacketPeerUDP.bind().

[KoBeWi]

String.plus_file() becomes String + "/" + file

This was actually renamed to String.join_path().

var the_callable = Callable(self, "__threaded_background_loader").bind([resource_path, thread_num])
start_success = new_thread.start(the_callable)

Or just start_success = new_thread.start(__threaded_background_loader.bind(resource_path, thread_num))

[FeralBytes]

FYI I have added many more notes to my previous comment to help with the conversion process.

[Calinou]

@FeralBytes Could you open a pull request to update some of the renames performed by the project converter? The code is here: https://github.com/godotengine/godot/blob/master/editor/project_converter_3_to_4.cpp

[aaronfranke]

This proposal has been implemented by godotengine/godot#51950