Lua via sol2

[olanti-p]

Summary

SUMMARY: Infrastructure "Bring back Lua"

Purpose of change

Experimental PR to bring back Lua modding.

According to DDA issue tracker and numerous discussions on Discord:

1. Having some scripting in game would be great. Not only it'd allow more interesting mods, it may also attract new modders with Lua background.
2. Porting Effect-On-Condition system from DDA 0.F+ will take a lot of effort due to sheer volume of changes and bugfixes. It's also a Domain-Specific Language, with all the issues that come from it (we'll have to manually expand language functionality, there are no libraries or tutorials, modders will have to learn it).
3. Bringing back Lua in Delta Fork by undoing its removal was a pain due to amount of code differences and time passed since then. Also, it resolves none of the issues of the old implementation.
4. The old Lua was "a PITA to maintain" due to multiple reasons:
   • Lua stack is hard
   • Lua build was fragile
   • You had to run a Lua script to generate bindings before compilation
   • You had to manually define function signatures and classes in a lua table and keep them up to date
5. Lua had little use in core, essentially turning it into deadweight and catalyzing bitrot. Part of it was from Lua being originally implemented as debugging API, part of it - from frequent build issues, part of it - from removal plans discouraging modders from sinking effort into new mods.

I can't magically make point 5 disappear, but the rest of the issues should be solvable.

As such, this PR uses completely different approach from the one that was used in DDA before Lua got axed.

The old Lua integration used Lua stack directly and then relied on scripts to generate C++ bindings file from a predefined Lua list that mirrored C++ definitions.

This version relies on a Lua C++ wrapper called sol2 to hide Lua stack management from the developer, so creating new bindings can be done by adding a few lines of human-readable C++. It still has to be done manually, but at least sol2 is able to automatically figure out types of objects being bound, so it's not much different from our de-/serialization code.

I'm not planning on re-implementing entire Lua functionality right off the bat, but this PR (if it goes in) should give a solid foundation to build on.

Describe the solution

The idea is to make Lua integration:

1. build and work consistently on all builds/platforms
2. require little to no additional build setup from dev side
3. be simple enough to be understood by people who haven't spent days and weeks studying Lua stack and virtual machine
4. be optional
5. be as nice as possible when it comes to using from Lua side

As such, the current solution is:

1. Add a folder in src named lua and put Lua 5.3.6 source code there. Lua is extremely fast to compile (it takes less than 10 seconds to compile on my machine when using single thread), so having it be built alongside other game source files and then be linked in one step mitigates a whole slew of issues (Lua version availability between platforms, Lua version mismatch between build machine and host machine, system Lua installation weirdness on some platforms)
2. Add a folder in src named sol and put sol2 v3.3.0 single-header source code there. Sol2 is a mature, feature-rich and performant C++ Lua wrapper that should cover many of our potential needs.
3. Add some automated tests (and Github Actions workflows) to help deal with compile- and run-time problems when it comes to building with Lua enabled.
4. Run an example Lua script on game start.
5. (Implemented in Refactor package loading, get rid of 'core' non-mod and per-world mod folders #2233) Refactor and streamline game package loading. Had to get rid of core non-mod (that's been causing trouble before btw, see Integrate basic.json into data/json #2106 ), and also of per-world mods (it's an obscure and almost unused mechanic with no access from UI).
6. On load stage, run Lua scripts provided by mods. The loading scheme is inspired by Factorio, see Data loading part in doc/LUA_SUPPORT.md for details.
7. Add debug Lua console. It can run simple commands, entire scripts and also shows log_ output from Lua functions
8. Add Lua hot-reload functionality by allowing to reload some of the mods' scripts on keypress & from menu
9. Add a bunch of bindings and hooks available for use by Lua mods
10. Implement a thin wrapper for registering usertypes dubbed luna. The purpose of this wrapper is to provide types registered with sol2 with additional runtime information such as type name as registered in Lua (sol only keeps track of type name as it's specified in C++, in cases where there is mismatch it becomes impossible to know Lua type from userdata) for (de-)serialization of Lua tables with userdata. It also emits additional information on members registered for the usertype, such as field names and types, method names and their arguments and return types.
11. Add a --lua-doc runtime switch that makes the game generate documentation for registered types based on information collected by luna. This way, we can have some rudimentary automatic docs generation for bindings that does not rely on some external tools.
12. Add a small Smart House Remotes mod that uses Lua to open/close curtains and garage doors in a house
13. Add a small SaveLoad Lua Test mod that uses hooks to preserve its Lua state throughout save/load cycle
14. In LUA=0 builds, show warning on mods that use Lua and disallow loading worlds that use Lua mods

Describe alternatives you've considered

Main alternatives and their disadvantages I listed in introduction.

Alternatives regarding current solution:

1. Use custom clang-tidy plugin to generate bindings.
   Unfortunately I don't know where to even begin writing one. I have little experience with clang-tidy, and the current tidy-with-plugins installation workflow could be somewhat hard for new contributors, see comment.
   Upd: turns out clang's syntax parsing can be used via a standalone library, so it should be possible to create a standalone bindings generator and just have the developers/contributors download precompiled releases.
   Upd2: same as with clang-tidy, it requires a lot of knowledge regarding clang codegen

2. Use some other Lua bindings library.
   I remember using LuaBridge a few years back, and sol2 is definitely nicer to work with. Maybe they've improved over the years, I didn't check. Also after some googling, sol2 seems to be the go to solution for C++ <-> Lua bindings in current year due to its overall performance, flexibility and C++iness.

Testing

Added PR workflows to build and test linux g++ and macos clang++ builds.

The tests include basic stuff like creating lua state, loading Lua libraries, bindings some definitions and running code.

• lua_class_members - Binds point class and its members, various constructors and methods (including math operators), then runs a Lua script that uses these bindings
• lua_global_functions - Binds a couple global functions and creature class family, then runs a Lua script that calls the global function and verifies that bound get_character_name accepts class avatar which is derived from Character from Lua side while original C++ function expects Character.
• lua_called_from_cpp - Runs a Lua script that creates and stores a function, then executes that function from C++ side and validates the results.
• lua_runtime_error - Causes runtime error on Lua side and checks that we receive appropriate exception on C++ side.
• lua_called_error_on_lua_side - Calls Lua's error() on Lua side and checks that we receive appropriate exception on C++ side.
• lua_called_error_on_cpp_side - Calls Lua's error() on C++ side and checks that we receive appropriate exception on C++ side.
• lua_called_cpp_func_throws - Throws exception from C++ function called by Lua and checks that we receive appropriate exception on C++ side.
• lua_table_serde* - Checks correctness of saving & loading a Lua table to/from JSON
• id_conversion* - Checks id conversion functions (raw pointer <-> string_id <-> int_id)

TODO list (this PR)

• Investigate possibility of writing automated bindings generator using libclang - Way to complex for me to tackle. Also, not sure if this is viable given how diverse the codebase is; it's way easier to write the bindings semi-manually.
• Expose more stuff to Lua
• Set up some Lua hooks
• Run Lua scripts provided by mods, figure out mod interop (I'm leaning towards copying Factorio's Lua loading scheme)
• Build a tiny but full-fledged mod showcasing Lua functionality
• Make Lua available in Windows builds
• Make Lua available in CMake builds
• Make Lua available in Android builds
• Look into semi-automatic Lua doc generation
• Update docs
• Gather feedback Feedback would have to be gathered after merging
• Figure out localization for Lua mods (may just end up making wrappers to gettext calls, xgettext supports Lua extract_json_strings.py can now parse Lua source and extract strings from relevant *gettext calls)
• Lua table (de-)serialization to/from JSON
• Handle usertypes and edge cases in table (de-)serialization (see TODOs in code)
• Look for a way to abstract away differences between int_id<T> vs string_id<T> vs const T* on Lua side - It seems sol does not allow customizing how const T* is treated for certain types. Added manual conversion functions.
• Workaround the sol2 issue where binding string_id<T> requires T to implement operators < and == - It would require either removing comparison operators from string_id<T> and int_id<T>, removing operator* and operator-> from them or patching sol2. I don't like all of these variants, meanwhile implementing comparison operators for the types is way simpler and doesn't kill ergonomics. Added a macro to make implementing them easier, it will have to do for now.
• Look for a way to split up catalua_bindings.cpp as it will get progressively slow to compile
• Introduce Lua API versioning and maybe game and mod versioning, to avoid potential mismatches here Ended up adding Lua API versioning in form of single integer, will have to see how it works out.

TODO list (long-term)

• Maybe a clang-tidy plugin to generate documentation from bindings? This would be easier than the current luna wrapper.
• Look for a nice way to bind typesafe coords
• Look for more options in mod interop
• Come up with naming convention for Lua side
• Expose even more stuff to Lua
• Add even more hooks
• Lua API testing infrastructure
• Write tests for API
• Make Lua mandatory and rewrite "content" parts of the codebase to use Lua (e.g. item use, player activities, examine funcs, more intricate parts of mapgen)

[Coolthulhu]

Maybe a clang-tify plugin to generate documentation from bindings?

I thought about using clang-tidy to generate bindings.
A good binding generator will certainly be needed to maintain Lua in long term. One of the reasons it died in DDA was the necessity to bind them semi-manually.

[olanti-p]

using clang-tidy to generate bindings

Tidy would be great, but I've given it some thought over the day, and I'm having trouble imagining how to make it work easily.

• The plugin itself - it'll probably end up quite complex, so whoever writes it will have to spend a lot of time learning llvm internals or have some preexisting knowledge of the topic
• To run the plugin we have to use a patched version of clang-tidy, which according to the doc has to be compiled from source for everything except Ubuntu Xenial
• There seems to be no instructions for running clang-tidy with cata plugins on macos
• Would the Lua build of the game require generating bindings? If so, does that mean macos won't have Lua builds?
• Only CMake solution has configuration for clang-tidy with cata plugins. What about Visual Studio and Make? Do we ditch them and migrate to CMake?
• If someone PRs a C++ change without testing how it'd affect bindings (e.g. someone on Windows who couldn't set up a custom build of llvm and has given up halfway - Lua is optional, so why bother if it kinda works), how do we test and fix it?

I'm not sure what else could be used as an alternative though.

One option I've considered is to have a PR job that runs a single custom tidy plugin that's basically a tweaked version of DDA's tidy serialize check that instead of checking whether member is serialized would check whether members and methods are bound to Lua. It would still be half-manual, but at least CI would point exact place where manual intervention is required. Another job could scrape the C++ files on upload branch for bindings (e.g. have it match every sol::state::new_usertype<T>(...) and sol::usertype<T>::operator[](...) invocations), and then build documentation as human-readable webpage and as machine-friendly text file that'll be diffed against previous version to see what API has been added/removed/changed.

[olanti-p]

Putting this on hold for now, clang codegen needs to be investigated + with #2250 we would have item manipulation API that's more consistent overall and more friendly to Lua in particular. Related: #2265

[olanti-p]

Okay, I've "investigated" clang codegen and discovered that thing is way over my head.

It basically has the same biggest issue as implementing generator via tidy plugin: someone would need to be familiar with libtooling, or spend some considerable time learning it, to be able to write such tool and extend its functionality. This is not something I would be able to pull off on top of everything else I already do for BN, not without quitting my day job.

[Coolthulhu]

The way I understand it, right now, it still requires manually binding C++ structs to Lua ones. Is that right?
It's OK for now, but it is not maintainable in the long term, so we will certainly need a better solution in the future.

[olanti-p]

Yes.
It is semi-automatic, as in sol handles actual Lua stack operations, argument validation and overload/inheritance resolution, and the luna wrapper handles doc generation for function arguments/return values and struct member types, but in the end you still need to manually define that "class Character has function i_rem that must be bound under name remove_item".

[Coolthulhu]

I can't get the smart remote to work. It always prints "no signal", regardless of whether I spawn it or find one spawned by mapgen.

[olanti-p]

It's a bit of a hidden mechanic, but the house it's linked to must have a grid with charged battery. It'll say "no signal" if there's no charge or the character is too far away.

[Coolthulhu]

Alright, got it working.
Does it have to be both mapgen-spawned and powered? If so, that's pretty much debug-only and the mod should probably be marked as obsolete.

[olanti-p]

Marked it as obsolete.

[scarf005]

Smart home mod

2023-08-28_21-29-48.mp4

Save/load

Docs generation

impressive.

[scarf005]

https://luals.github.io/wiki/annotations/

by the way, in the future generating definition files would be great, as their syntax doesn't differ much from lua docs generator.

here's a branch with incomplete POC definition file generator: https://github.com/scarf005/Cataclysm-BN/tree/lua-definitions