Prototype zig package manager integration

[ifreund]

Upstream zig recently added support for a "system package mode" intended to integrate the package management with distro packaging. This PR uses these new features to implement a proof of concept for packaging/distributing zig packages using chimera's cbuild toolchain.

This MR successfully packages waylock's 1.2.0 release and its dependencies. Native compilation on x86_64 and cross compilation to aarch64 both work.

I've opened this MR to encourage collaboration between chimera's maintainers and upstream zig maintainers with the goal of finding how we can work together to ensure our respective end-users are best served. I personally have a foot in both camps being both a zig core team member and chimera user/contributor.

Here are the main issues I see with the current state of the zig package manager from a distro packaging point of view:

1. Since zig identifies packages by their hash, there is no way for a distro to bump a zig package from e.g version 1.0.0 to 1.0.1 without patching the build.zig.zon of all consumers of that package.
2. There is no easy way to obtain the hash of a zig package using the cli without copying into the global cache. It should be trivial to add an option to zig fetch for this purpose which would allow full automation of the install step of the zig-wayland and zig-xkbcommon templates for example. Possibly this would be made irrelevant to a solution to 1. however.
3. There is no easy way to delete the files in a zig package that are not referenced by the build.zig.zon and included in the package hash. There is no reason for distros to include these files in the packages they redistribute. A more generally useful solution might be to add a CLI command to list files included in the package, which the distro packaging tooling could use as a whitelist for what to include without needing to semantically parse the build.zig.zon. Edit: zig fetch does this, I just misunderstood how to use it.
4. While --host-target, --host-cpu, and --host-dynamic-linker were added, what toolchains like cbuild which support cross-compilation really need are --target, --cpu, and --dynamic-linker which describe the target system, not the host. This PR currently uses the -Dtarget and -Dcpu flags defined by waylock's build.zig instead but this is technically not portable.

[nekopsykose]

half of the WIP for zig itself (the other half being the build style you've now implemented) is that currently we don't really have plans to have multiple llvm versions at once. the current unwritten requirement for things linking libLLVM (zig also uses libclang and liblld but that's just fluff) is that they are generally ready by the time .3/.4 is out or so (when we upgrade to new majors). so far, zig isn't up to date with the latest llvm in a shipped release near these points (though i do see that on master it happens very fast). not sure how this is best resolved though. maybe eventually we'll just have to have multiple versions..

[andrewrk]

There is another possibility that we can explore, which is to take advantage of zig's C backend, and support for building from source without LLVM.

Such a build of zig is not suitable for direct users of zig, however, it could be used by the distro to package zig applications by compiling them to C code and then compiling and linking that C code with the system toolchain.

[nekopsykose]

hmm.. if that actually works for all regular zig projects one would want in a distro-packaged way (river, ncdu2, ..) then it honestly doesn't seem like the worst idea :D. i had no idea it was possible

[ifreund]

I haven't tried that approach yet but it certainly sounds promising in theory. I suspect there may be some rough edges that need to be ironed out but that's why I opened this PR in the first place :)

[nekopsykose]

re 1 & 2: this also has the effect that the install_files needs a name="some-hash" that makes updates require changing one more line. but yes, a solution to 1 probably makes that easy too since then it could be named by a more stable {pkgver}

aside from these passing comments this looks really nice! afaict this is kind of the way the hare language chose to approach third-party modules too?

[andrewrk]

Thanks for putting this together!

Since zig identifies packages by their hash, there is no way for a distro to bump a zig package from e.g version 1.0.0 to 1.0.1 without patching the build.zig.zon of all consumers of that package.

Two points on this one:

1. The distro can choose to lie and provide whatever version they want for any particular hash, swapping one version for another. The zig build system by design only verifies hashes on fetch, and if the system provides a particular hash, it will be trusted and used.
2. An upcoming feature that will not land in 0.12.0 will address the general use case of multiple package versions existing in a dependency tree, and the ability to swap one package version for another. Probably at the same time this will come with CLI flags or other functionality that makes it easy for a distro to participate in this process.

There is no easy way to obtain the hash of a zig package using the cli without copying into the global cache. It should be trivial to add an option to zig fetch for this purpose which would allow full automation of the install step of the zig-wayland and zig-xkbcommon templates for example. Possibly this would be made irrelevant to a solution to 1. however.

Would you mind writing up a zig issue for this one? I started working on it, but I found myself not able to explain the problem very well. In particular, what would you want to happen to the directory of files that was fetched in order to calculate the hash? I just need to understand the use case a bit more in order to solve it satisfactorily.

There is no easy way to delete the files in a zig package that are not referenced by the build.zig.zon and included in the package hash. There is no reason for distros to include these files in the packages they redistribute. A more generally useful solution might be to add a CLI command to list files included in the package, which the distro packaging tooling could use as a whitelist for what to include without needing to semantically parse the build.zig.zon.

Only files that are referenced by the build.zig.zon are included in the package hash. When you use zig fetch to fetch something, it does delete everything not referenced by the paths field of build.zig.zon.

If it helps, we can certainly provide such a tool. I'm not sure I understand the workflow yet though. Is this accurate?

1. Distro starts by unpacking pristine upstream source tarball
2. Distro deletes files not referenced by build.zig.zon
3. Distro obtains zig package hash
4. Distro creates a directory named after the package hash with the remaining upstream source files
5. That directory plus some other metadata becomes the distro package

And then this point has to do with streamlining step 2?

While --host-target, --host-cpu, and --host-dynamic-linker were added, what toolchains like cbuild which support cross-compilation really need are --target, --cpu, and --dynamic-linker which describe the target system, not the host. This PR currently uses the -Dtarget and -Dcpu flags defined by waylock's build.zig instead but this is technically not portable.

Hmm, I see your point. I'm not sure how I got fixated on this idea of overriding the host. In the face of this new use case, it seems like I went down a completely wrong path.

[ifreund]

1. The distro can choose to lie and provide whatever version they want for any particular hash, swapping one version for another. The zig build system by design only verifies hashes on fetch, and if the system provides a particular hash, it will be trusted and used.

Good point, however this approach would not be sufficient to solve the situation where packages A and B list two different versions of package C (say 1.0.0 and 1.0.1) in their respective build.zig.zon files. I don't see a good way around patching the build.zig.zon of either A or B in that case.

From the distro point of view, a possible fix for this would be for --system foobar to look for packages in the foobar directory by name rather than by hash. This would rely on the zig ecosystem using consistent naming for packages, with the distro patching inconsistencies/duplicate names. It's currently possible (and I'm pretty sure an intentional feature) for the consumer of a zig package to totally ignore the name the package gave itself and assign it any arbitrary name, I'd hope this feature wouldn't be abused however.

2. An upcoming feature that will not land in 0.12.0 will address the general use case of multiple package versions existing in a dependency tree, and the ability to swap one package version for another. Probably at the same time this will come with CLI flags or other functionality that makes it easy for a distro to participate in this process.

It sounds like this would in theory resolve this issue in the long term, I wonder about how this would be implemented however as it seems that it would require reliance on consistent package naming or some other new metadata.

Would you mind writing up a zig issue for this one? I started working on it, but I found myself not able to explain the problem very well. In particular, what would you want to happen to the directory of files that was fetched in order to calculate the hash? I just need to understand the use case a bit more in order to solve it satisfactorily.

Only files that are referenced by the build.zig.zon are included in the package hash. When you use zig fetch to fetch something, it does delete everything not referenced by the paths field of build.zig.zon.

Actually, I think I misunderstood how zig fetch works and think that zig fetch --global-cache-dir /usr/src/zig/packages/ foobar would actually do exactly what we want here. I'll give it a try :)

Hmm, I see your point. I'm not sure how I got fixated on this idea of overriding the host. In the face of this new use case, it seems like I went down a completely wrong path.

To be fair, most distro packaging toolchains don't support cross compilation so it's easy to look at a few and assume they only care about targeting the host system.

[ifreund]

Pushed a commit leveraging zig fetch to avoid hardcoding the package hashes in the templates. It feels a bit like an abuse of --global-cache-dir but perhaps it's fine.