Allow for linking from GDScript to functions in C-compatible binary objects, as Resources

[MichaelMacha]

Describe the project you are working on:
I am currently working on a web-based metroidvania with a fairly complicated dynamic map.

Describe the problem or limitation you are having in your project:
GDScript is great for basic code, but tends to fall short for broader tasks. I recently went through a number of issues when attempting to subclass my map grid to use a mask; and finding that alterations had to be made to my original Grid class to get it to work. This would be OK for simple things—Bethesda's Papyrus isn't much better, nor is Lua or Squirrel—but I found myself wishing I could simply build this specific feature in another language and link it in…

Describe the feature / enhancement and how it helps to overcome the problem or limitation:
When I use D, I can link to functions declared in any C-compatible assembly. This is very helpful when I need to interface with C, C++, or any other C-compatible language and don't want to rewrite the code or build a massive proxy for it. (I build onto my home LAN server like this all the time, it's pretty easy.)

It would be great if GDScript could also call functions from compiled, but not linked, objects (the *.o or *.obj files spat out before the executable is) that are C-compatible. They could be loaded as a type of built-in Resource. As GDScript is an interpreted language, not an actual machine binary, I imagine this would simply involve an internal method to find the function by name and pass data to it. This would expand Godot's wonderful language-agnosticism quite a bit, and would allow for many new avenues of improvement.

Additionally, when working on an algorithm that requires, or benefits from, a more full-featured language, virtually any systems language could be used. Moreover, occasions when the interpreted-language slowdown becomes a critical problem could become optimized with it; though I know this isn't common. (This could also, in theory, occur for desktop game development.)

Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams:
Suppose we had a basic C program. I'll borrow one from Dlang's documentation.

#include <stdio.h>
void say_hello(void) 
{
    puts("Hello from C!");
}

Compile this to an object file; we'll call it hello.o. Store it in a Godot project as a resource.

In GDScript, load this object as a Resource subclass.

var hello = preload("res://objects/hello.o")

Call the function with an internal method for finding functions by name. Something like this.

hello.get_function("say_hello(void)").call()

And accordingly, we would get puts("Hello from C!") executed and printed out on Godot's (or the game's) stdout, if it has one.

If this enhancement will not be used often, can it be worked around with a few lines of script?:
I have my doubts. Godot has a number of reasonable restrictions against making system calls from WebAssembly, as do modern browsers. For desktop stuff, it could maybe be done with some piping but I'm not convinced it wouldn't still require more than a few lines and potentially extra dependencies, along with testing on each target platform. For WebAssembly, it may actually require a web request (like AJAX did), which opens up all kinds of complications. If these objects could be treated as resources, code-as-data-style, it would get around all of this.

Is there a reason why this should be core and not an add-on in the asset library?:
If it can be made as an add-on asset, that would work for my purposes, as long as it could successfully be integrated with an HTML5 game. My vision is that in the end, any action of the linked object would appear to be a native decision by the game, including any permissions it might need.

[Zireael07]

as long as it could successfully be integrated with an HTML5 game.

That is a big blocker in this proposal, I believe.

[Calinou]

This sounds very similar to GDNative, except your proposal is about using non-linked compiled objects. (I've never heard of that approach before.)

[groud]

Yes, have a look to GDnative, it should already cover most of your needs.

[Xrayez]

NB: compiled object files on Windows have an extension of obj, which would be wrongly recognized and imported as the 3D object models, see also godotengine/godot#28343. 👀

[MichaelMacha]

It's my understanding that GDNative requires a dynamic library and doesn't work with WebAssembly or most phones. If this is incorrect, boy am I missing forests for trees.

[MichaelMacha]

@Calinou It's definitely possible; you just have to separate the compilation phase from the linking phase. For D, you would use something like (from their very-same documentation):

gcc -c chello.c
dmd hello.d chello.o

With the D line:

extern(C) void say_hello();

and D will call the function without argument. It does seem a little strange coming from single-language builds, but I do it all the time these days when I've got some old, working, code using C, but am building with D. Or, if I have some other language I want to interface D with as a binary. There's no difficulty at all, it's basically just static linking of an object from another compiler.

The only thing you really need to watch for (or so I've been told) is the name-mangling system that C uses, and ensuring that your own compiler or interpreter is familiar with it, but that's really well documented at this point. I don't imagine it would be that different from an interpreter instead of a compiler.

[MichaelMacha]

@Zireael07 Would you mind discussing how that's a blocker? I can't say I'm too surprised, I know WebAssembler is a unique platform; but if the external objects were built for that platform, would it be doable?

With D, again as an example (though it could easily be something like C or Go), I build webassembly with a cross-compilation like so, given that I'm using LDC.

ldc2 -mtriple=wasm32-unknown-unknown-wasm -betterC my_code.d

and, this produces a .wasm file that will run in any modern browser, on any half-decent operating system, with the right JavaScript interface. I'm not 100% on this but it's my understanding that wasm files are basically binary objects built for the web, at a relatively slight performance penalty.

I'm willing to look into this myself and do some digging around, but I've currently got not-just-the-game but also a couple of books I need to finish writing and get up on Kindle. Once my schedule is clear—maybe a couple of months—it could be a fun project this summer. I haven't really rooted around in the Godot source yet, so I'm still guessing a bit.

[Zireael07]

I think it's a blocker for you (quote:

that would work for my purposes, as long as it could successfully be integrated with an HTML5 game

in the same way there's no WebAssembly support yet for GDNative, so you'd be limited to desktop platforms.

[follower]

A couple of related thoughts:

• In terms of "C-compatible binary objects", based on the example given in Allow for linking from GDScript to functions in C-compatible binary objects, as Resources #830 (comment) isn't what you're wanting support for .wasm files not e.g. .a or .o files? (i.e. you're not literally wanting a web browser to load a .o file, you're okay if you have to compile to .wasm?)

• When a Godot app is exported to the web it's possible to call out to Javascript: https://docs.godotengine.org/en/stable/getting_started/workflow/export/exporting_for_web.html#calling-javascript-from-script

  This means (in theory, I haven't tried it) you could customise the export page template to also load the .wasm files you wanted to use and call into them with the browser's usual WebAssembly functionality.

• In terms of support for desktop and similar functionality:

  • it seems like using a dynamic library would be easiest, in which case Foreigner enables that for Godot via a Foreign Function Interface (although I would like to see in exposed in core--the actual dynamic loading functionality is already implemented for GDNative it's just not exposed & rejects non GDNative libraries; it also doesn't support the FFI aspect): https://github.com/and3rson/foreigner

  • If the static linking aspect is important then...I guess you'd need to do something with a module & custom engine compiling/linking? (Although, maybe libffi would help in this case too?)

• In terms of desktop support for WebAssembly I've written about I project I've released that can enable that here: Add WASM (WASI) host support (including, but not limited to, the HTML5 target) #147 (comment)

[CsloudX]

I often work with 3rd party dll, there was only *.dll file(no *.o file), but all of them was standard c dll. So I wish GDScript can load standard c export dll without any other.
can use like this:

var dll = load_dll("res://dll/a.dll") # any valid path
var rs = dll.call("add", 10, 20) # in DLL, method was  int add(int a, int b)

[YuriSizov]

Godot already provides means for integrating DLLs into your projects. In 3 it was GDNative, as mentioned above, and in 4 it's GDExtension.

[Calinou]

What @CsloudX is looking for is FFI, which is already proposed in #1917.