Support of Zig versions >= 0.13.0? How hard will it be?

[Sh4pe]

I started an attempt to make pydust compatible with Zig 0.13.0. For this, I created a tiny playground project that uses pydust via build.zig.zon with the goal to have a tiny Python module. I've run into several issues that appear to be easy to fix. I've added some patches on top of develop at 1908fe8 which you can find in patches.tar.gz and also in my fork. However, the next error turns out to be complicated:

# in my playground project
sh4pe@sh4peux ~/D/d/z/luatable_play (main)> zig build
modules: 1 { pydust }
len toplevel steps deps: 0
install
└─ install luatable_play
   └─ zig build-exe luatable_play Debug native 1 errors
/home/sh4pe/Documents/devel/zig/ziggy-pydust/w/feature/zig-0.13/pydust/src/discovery.zig:45:16: error: type capture contains reference to comptime var
    break :blk struct {
               ^~~~~~
referenced by:
    State: /home/sh4pe/Documents/devel/zig/ziggy-pydust/w/feature/zig-0.13/pydust/src/pydust.zig:15:39
    rootmodule: /home/sh4pe/Documents/devel/zig/ziggy-pydust/w/feature/zig-0.13/pydust/src/pydust.zig:44:10
    remaining reference traces hidden; use '-freference-trace' to see all reference traces
error: the following command failed with 1 compilation errors:
/nix/store/f2119p2rpq7k67klrrmlmrg3z1zjzv3i-zig-0.13.0/bin/zig build-exe -ODebug --dep pydust -Mroot=/home/sh4pe/Documents/devel/zig/luatable_play/src/main.zig -Mpydust=/home/sh4pe/Documents/devel/zig/ziggy-pydust/w/feature/zig-0.13/pydust/src/pydust.zig --cache-dir /home/sh4pe/Documents/devel/zig/luatable_play/.zig-cache --global-cache-dir /home/sh4pe/.cache/zig --name luatable_play --listen=-
Build Summary: 2/5 steps succeeded; 1 failed (disable with --summary none)
install transitive failure
└─ install luatable_play transitive failure
   └─ zig build-exe luatable_play Debug native 1 errors
error: the following build command failed with exit code 1:
/home/sh4pe/Documents/devel/zig/luatable_play/.zig-cache/o/4e7f29dfea32ac791bb5b3d3a18e48b1/build /nix/store/f2119p2rpq7k67klrrmlmrg3z1zjzv3i-zig-0.13.0/bin/zig /home/sh4pe/Documents/devel/zig/luatable_play /home/sh4pe/Documents/devel/zig/luatable_play/.zig-cache /home/sh4pe/.cache/zig --seed 0x96d09050 -Z000f44489101f0c3

This seems to be related to the comptime memory changes introduced in Zig 0.12.0. To me, it looks like the usage of comptime vars is integrated very deeply into State inside discovery.zig. This bears the question how much of a redesign that would need in order to work with Zig 0.13.0. Or will it work at all?

So what are your plans and ideas for support of versions >= Zig 0.13.0?

[gatesn]

Yep, I think you've struck on the main problem here which is our (ab)use of comptime.

We're going to have to figure out a new approach and API design for pydust. The reason we make use of comptime state is to "register" to existence of types, e.g. py.class(struct { ... }). This function actually returns its argument, meaning the struct you register is the struct you use and all functions on it stay around.

If Zig allowed us to create struct types with custom declarations (pulling the functions out of the user struct and into a new pydust struct) then that would be great. But it doesn't...

So I think we might need to get creative.

I'm going to turn this issue into a discussion of approaches we might take.

[Sh4pe]

Alright, thanks for the reply. I don't know if I can help at the moment since my Zig skills are not very well developed and the knowledge of pydust internals is even more limited. If I should come up with some idea to move this forward, however, then I'll share it here.

Cheers!

[rupurt]

@gatesn really appreciate the work you've put into this. I know it's a long way off... But I'm going to make understanding Zig/C/Python interop my advent of code project. Hopefully I will be able to contribute once I have a good level of comprehension.

If you're strapped for time it might be worth coming together as a community to figure out how we can market this project for advent of code to get a lot more people involved. I feel like first class Python tools would be a huge accelerant for Zig. Maybe @kristoff-it has some ideas?

[gatesn]

That sounds like a great idea. I'd be happy to jump on a call / meet up in London to discuss some of the design decisions and ideas we have on how to fix/improve pydust.

[ivansantiagojr]

Hey! I am starting to advance my studies in Zig and Python internals and I would love to contribute to Ziggy Pydust (which I really like). Is there any recomendations on where to start studing and analyzing Ziggy Pydust, @gatesn? I explored the code base a little, but I confess I do not really know where to look first.

Also, how is the journey on studing Zig/C/Python, @rupurt? I would love to help!!

[robert3005]

You have https://github.com/spiraldb/ziggy-pydust/blob/develop/pydust/src/pydust.zig#L43 as a top level method the user would call which then mutates this state struct to register a new type https://github.com/spiraldb/ziggy-pydust/blob/develop/pydust/src/discovery.zig#L35. When it comes to produce the python defs we start with a module https://github.com/spiraldb/ziggy-pydust/blob/develop/pydust/src/modules.zig#L32 which has slots https://github.com/spiraldb/ziggy-pydust/blob/develop/pydust/src/modules.zig#L34 and methods https://github.com/spiraldb/ziggy-pydust/blob/develop/pydust/src/modules.zig#L35. Slots in turn have attributes https://github.com/spiraldb/ziggy-pydust/blob/develop/pydust/src/modules.zig#L81 and that has types https://github.com/spiraldb/ziggy-pydust/blob/develop/pydust/src/attributes.zig#L51