Turn scripts into dynamically created GDExtension classes

[kisg]

Update 2023-10-01: This is a pretty radical proposal that essentially proposes the removal of the scripting extension mechanism from the engine (where you can attach scripts to Godot Objects). At the time of this update, this change is simply not feasible: There are valid use cases for both GDExtension classes and for the attached scripts.

We have a new proposal #7950, that is better suited for the engine at this time.

It proposes a purely opt-in solution for defining GDExtension classes in GDScript.
Other GDExtension-based languages already allow this by default, and most likely C# will also be migrated to a GDExtension language.

Describe the project you are working on

Godot in a commercial AR project from where we intend to contribute our enhancements.

Describe the problem or limitation you are having in your project

The current way scripts are working unnecessarily complicates engine internals. There are currently 2 ways to extend the functionality of a Godot type:

• Use the single inheritance relationship in the type system to define a subclass.
• Attach a single script to an object instance that takes over and extends / replaces some of the functionality of the class.

This complicates the engine, the code is full of blocks like this:

if (get_script_instance()) {
...
}

When modifying engine classes, extra care is necessary to keep the script code paths intact. Script calling code paths tend to be slower / more complex than regular GDExtension code paths. (I am not talking about the GDScript side, just the part until GDScript gets called).

I understand that attaching a script to an object instance is more a "composition" relation than an "inheritance" relation, but it is a very limited composition that (in my opinion) creates more problems than it solves.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Change script language implementations (e.g. GDScript and Visual Script) to dynamically register a GDExtension class for each of the scripts created. Each script will essentially create a new subclass of a Godot Object that can be instantiated the same way as built-in classes. It will also be possible to implement a class in GDScript / Visual Script, and then subclass it from C++ or from any other language.

This change will simplify the engine code and provide a single solution for extending the engine with new code: the GDExtension API.

If the proposal is accepted, we are willing to contribute the implementation for this feature (targeting Godot 4.0)

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

The main UI change would be, that script resources no longer have to be bound to a node. When a script resource is created, it automatically creates a new node / object type, that can be used from other scripts and also added to the scene tree as if they were builtin classes or "normal" extension classes.

This requires a bit of education of the engine users, but the UX can be made very clean and easy to use in the editor:

• when a new node is added in the naming dialog there would be a checkbox to extend the class and specify the language to be used for the extension
• the change node type context menu could change the base type for the script
• double clicking on a node that is backed by a script could bring up the editor for the underlying script automatically.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No, it is a core change in the engine

Is there a reason why this should be core and not an add-on in the asset library?

It is a core engine change.

[kisg]

I think this issue shows the complexity of handling scripts (in this case with regards to dynamic properties): godotengine/godot#43491

This is one of the reasons why I think this proposal would improve the overall usability of Godot.

[YuriSizov]

I think this issue shows the complexity of handling scripts (in this case with regards to dynamic properties): godotengine/godot#43491

_get_property_list() is a core mechanism for dynamic properties in Godot. Engine uses the same system internally. GDExtensions would use the same system if they need dynamic properties. Usability for user scripts, such as adding property groups for exported members, is not a GDExtension problem, it's a GDScript syntax problem. We just need to add annotations for grouping exports (see #1255 (comment)).

[kisg]

@pycbouh Yes, _get_property_list() is the standard hook used in the Engine. But if you take a look at here godotengine/godot#43491 (comment) (and the comments below), you will see that the handling of scripts complicates this by introducing get_script_property_list(), and the special handling around it. This is not at all a GDScript specific issue. Yes, it is possible to make it work, but in my opinion, it needlessly complicates the engine, especially with the new GDExtension API / ABI.

[MagdielM]

While I personally really like this proposal and find working with plain ol' types much more intuitive than scripts as extensions, you have to keep in mind this is a radical shift in workflow for Godot, which may make migration to 4.0 even harder for the average user. I'm also not sure how this would affect Mono support (big C# guy here).

[kisg]

@MagdielM Thanks for your comment.

C# support actually would become more clean, the distinction between scripts vs GDExtensions would disappear. You would simply create a subclass of any Godot class in C# and that class in turn could be subclassed in any other language (it could be GDScript, C++, Rust ... etc.). The only thing that you have to keep in mind, that only the parts exported over the GDExtension API will be visible from other languages.

This is actually very similar to how C# and .NET works: C# is a much more rich language than what the Common Language Specification / Common Type System of .NET describes: from other languages only the CTS compatible parts of your C# classes are accessible.

I don't think that there would be a big learning curve for most users. Current GDScripts would not have to be modified at all (or only a very minimal addition would be necessary to allow type names that are not computed from the file path / file name). The only difference would be that when they want to use the script they don't attach it to a node (that their script extends), but they simply change the type of their node to the type defined by the script.

[AnidemDex]

The only difference would be that when they want to use the script they don't attach it to a node (that their script extends), but they simply change the type of their node to the type defined by the script.

Doesn't this means that, everytime that you create a new GDScript, you're creating a new type of node instead? Don't you end with... many nodes at the end? What about other Object types like resources?

[JoNax97]

I don't see that as a bad thing at all. You will end up with many scripts with the current approach anyways. As for other Object types, I guess you would have to declare what type are you extending from (Node could be the default, to keep compatibility with the current system).

[kisg]

Exactly as @JoNax97 writes: in effect the scripts we create now also create new Node subclasses.

This proposal would also allow the Godot users to extend other types in GDScript as well (like Resources, or anything else that the GDExtensions system supports), and if the WASM based AOT compiler for GDScript is implemented (#3370), it would even allow near-native performance for these new types.

[TechnoPorg]

This would simplify things a lot, for example in something like godotengine/godot/pull/48201. One concern I have, though, is that this might require the Godot editor to have a C++ compiler available. That would either make Godot setup harder for the average user, if they provided it, or it would make Godot binaries a lot larger, if the compiler were shipped with the editor.

[touilleMan]

following #3927 (comment)

I think this proposal is interesting, however Godot 4 is currently in alpha so things are moving and it may already be too late to introduce such a complex change...
On the other hand this change breaks how scripts works so once Godot 4 has shipped it's pretty sure it won't be possible until Godot 5 😨

So it would be good to have feedback (@akien-mga I summon you 😄 and @godotengine/gdnative ) without delay from the core team on how realistic this proposal is, so basically:

• no way
• it will be considered if a POC is provided fast enough
• there is no hurry Godot 4 is far from ready to ship

I'm especially interested on this answer to have an idea on how I should start working on #3927

[touilleMan]

After looking a bit more on how GDExtension is implemented, thinks seems simpler that expected ;-)

Extension is implemented in a very similar way than script is:

• the Object class has a script_instance and an _extension attributes.
• Each node defined in Godot (e.g. Node3D, KinematicBody etc.) inherit of Object
• when using a .gd script file, a script instance is created and added a attribute to the Object instance representing the main type
• similarly when instantiating an extension classe, the base native Object is first instantiated, then the extension object is connected to it as _extension attribute
• when doing property lookup, Godot has to search in the script, then in the extension and finally in the base class (see https://github.com/godotengine/godot/blob/0217d44dfde8fcac90cc2f9c1f38a9b37042bb7e/core/object/object.cpp#L457-L476)
• similarly when a method lookup is done, Godot has to search in the script, then in the extension and finally in the base godot class. However extension methods are also registered into ClassDB (just like godot class methods are) so there is a single lookup for extension/native class method (see https://github.com/godotengine/godot/blob/0217d44dfde8fcac90cc2f9c1f38a9b37042bb7e/core/object/object.cpp#L379-L398)

So it seems pretty doable to just remove the script_instance attribute and use instead _extension for every usecase.
Of course this would require to add some methods that are needed by script language (e.g. property_get_fallback).

[kisg]

Hi @touilleMan, thank you for looking into this.

Your findings confirm what I saw when I looked through the code. :)

[neikeq]

Keep in mind, there are many issues with GDExtension. From very serious ones like godotengine/godot#57427 to usability regressions (mainly due to no file relationship). Something like this can't be done until those are solved.

[Zireael07]

@neikeq: Speaking of issues, my use case is speeding up tool scripts, IIRC those were (and still are) problematic for any non-GDScript languages?

[neikeq]

@Zireael07 Other languages may have usability issues. The only one I know for C# that is not a bug is that you need to build first to be able to enable the plugin (GDExtension libraries will likely be prebuilt for distribution, though). You could also mix GDScript and GDExtension native code to get the best from both.

[the-ssd]

@Zireael07 LLWM isn't that big

LLWM source code without tests is only 126.7 MiB
that is a lot compared to Godot's 75 MiB but if its optional like export templates which are 964 MiB (500-600 MiB compressed)
and you can cut out a lot of code like RISC-V target and PowerPC target

i think LLWM is the way to go

[Zireael07]

@SSD-slav doubling the size of the core is absolutely not an option (and I can't see how this can be made "optional", unless you mean it should be an export only feature like converting text to binary on export, but I believe even this use case would need having llvm code in core)

btw you replied on the wrong issue

[the-ssd]

@Zireael07
i mean optional like a check box when exporting and warning that this is going to be installed
maybe it can be in Editor > Manage Export Templates... or Editor > Manage Editor Feathers...

[ywmaa]

I was thinking, would this also allow inheritance between different languages ?

GDscript / Visual script / c#

[Frontrider]

I was thinking, would this also allow inheritance between different languages ?

That is a basic feature of webassembly.

[Shadowblitz16]

This is basically kinda what I wanted with my true oop workflow suggestion...
I guess I just can't communicate well

[kisg]

Check out our follow-up proposal that is less radical: it is a purely opt-in feature to allow GDExtension classes to be defined in GDScript.

#7950

[BenMcLean]

This seems like a way better long term plan. If Godot is not going with this, could someone explain why?

I understand that GDExtension might still have ongoing issues which prevent unifying the interface for now, but it seems like having one unified interface should be a long term goal for Godot in the future.