implement libc in zig

[andrewrk]

If Zig ships with a libc implementation, it seals the deal for streamlined cross platform compilation. It improves the situation for native compilation as well. When a zig project depends on a C library such as GLFW, one could package it up with zig build system to build with zig (See #490). This accomplishes several things:

• A single zig build from the root project on any platform will fetch the project's dependencies and perform a deterministic build using only zig and no system dependencies.
• Zig can be intelligent about how it links dependencies together. It can link a bunch of .o files together instead of doing dynamic linking, where only 1 of the .o files has the libc implementation in it. So any code shared by zig std library and the libc implementation is actually shared in the binary, including stdio buffers and such. The default allocator can be shared between zig std lib and malloc/free.
• This build would have no native dependencies, so cross compiling works perfectly out-of-the-box, even though you have C library dependencies that depend on libc.

Our goal is for developers to reach for zig instead of gcc, clang, or msvc, even if they intend to use it only to compile C code.

For linux, we can port musl. A lot of our stdlib is ported from there anyway - a lot of this code would be shared between zig standard library and the libc implementation. We should also look at glibc, for example for the getpwnam function, which in musl reads /etc/passwd but in glibc has an extension for a more advanced user account system.

For windows it might be worth looking into newlib-cygwin, reactos libc, and wine libc.

For macos, we always link dynamically against libSystem which has libc in it because this is the stable syscall interface. When cross compiling we allow functions to be unresolved that come from libc. We still need to ship .h files with appropriate constant values, however. I think the .h files from apple might be BSD licensed and thus we could redistribute those.

[pluto439]

On windows, from what part of libc should people start?

[andrewrk]

Can you elaborate on the question?

[pluto439]

Do you need all libc, or only some parts of it? Which parts are more important right now?

[andrewrk]

The goal is to provide API compatibility for existing C projects. We can let that drive the use case. So to figure out what is more important, we would try to, for example, build ffmpeg against a zig-libc.

[pluto439]

I'd start with libjpeg, but I need exceptions for that. Or at least setjmp/longjmp.

[andrewrk]

You can implement setjmp/longjmp with inline assembly.

[tiehuis]

For anyone interested in this, I've started a simple repository which should give you some ideas here. Anyone reading, feel free to use this as a base or as inspiration.
https://github.com/tiehuis/zligc

I won't be doing much implementation here (too many things I already have to work on) but I may use this to figure out some zig -> c questions and some missing build system details needed in zig to accomodate the required goals.

[suirad]

@andrewrk would you mind making a zlibc repo under the ziglang account that people could coordinate on to accomplish this? You could probably start it with a fork of @tiehuis' repo if you want.
I know you probably already have plenty to manage with zig itself, so you could probably recruit a mod or two to manage the repo.

[andrewrk]

@suirad sorry, I missed your comment above. The libc code is going to live in this repository itself; contributions can be done via pull requests. I'll get the effort started so that people can see the pattern.

Now that we have #490 done, this issue of having libc is even more valuable, and easier to implement. How it's going to work is that Zig will have a set of supported libcs corresponding to a target.

• --target-os linux --target-environ gnu (or other gnu environs) will be glibc, and it will be a dynamic dependency, the only kind supported by glibc. Zig will be able to build glibc's startup files from source (crtn.o, crtbegin.o, etc) and provide .h files. Zig does not need implementations of libc functions since it is dynamically loaded.
• --target-os linux --target-environ musl (or other musl environs) will be musl's .h files. It will be a static dependency. Some of musl's source code can be imported directly such as their startup files. Other implementations of libc functions will be shared with the Zig standard library - this will prevent code bloat. For example the math functions will share implementations.
• --target-os windows --target-environ msvc I think for this one we will use newlib/wine source code. It will be a static dependency.
• --target-os windows --target-environ gnu (or other gnu environs) - this is mingw libc. Same idea as musl. Static dependency. Build from source.
• --target-os freebsd (not sure what the default target environ is) - this is similar to glibc for linux. Zig will be able to build the startup files from source, but the implementations will be resolved at runtime.
• --target-os macosx -libSystem actually does not even require any startup files. Brilliant.
• --target-os netbsd same idea as freebsd

Once this is done, it raises the bar for Tier 1. A target can only be Tier 1 if Zig can cross-compile link against its libc.

Zig will no longer look for the native libc and try to build against it. zig libc will be the only way to do that. And unless a --libc foo.txt argument is provided to use a custom libc, Zig will use one from the bulleted list above; never the native one. This makes Zig 100% free of system dependencies, which makes it a true cross compiler.

[andrewrk]

I've done a proof of concept of this with glibc and it's merged into master. (The CI build is broken but I have a fix coming shortly)

There's a new tool libc/process_headers.zig and a wiki page documenting how to maintain the various libcs.

Here are some screenshots on Windows, Linux, macOS:

So at this point it's a checklist:

• musl libc - this one we will do the headers thing, but try to implement most of the functions in Zig. Some of the functions we can build from C source - that's fine. We can always do a piecemeal port.
• Windows libc (newlib? wine?) - same idea as musl - get the headers from somewhere, but the implementation should be shared with musl implementation, with perhaps some .c code to augment it.
• mingw libc / cygwin libc - these will probably be their own issue. Probably low priority. Why target these when you can target Windows directly?
• macosx libc - we don't need any startup files for this one, but we do need headers.
• FreeBSD libc - follow glibc's example
• NetBSD libc - follow glibc's example
• OpenBSD libc - follow glibc's example

[suirad]

There probably be a new builtin.<libc-implementation> or something exposed to allow switching implementations at compile time. That should be added to the checklist.

The reason why it should be exposed and not assumed based on OS, is because some platforms are able to use multiple types, such as linux with musl & glibc.

[andrewrk]

There probably be a new builtin.<libc-implementation> or something exposed to allow switching implementations at compile time.

That's @import("builtin").abi, which is specified at compile time with -target <arch><subarch>-<os>-<abi>.

[andrewrk]

Musl is done

Repeating this here:
Currently zig builds musl from source with minimal modifications. The next step will be to also include std/special/libc.zig (which I have not started yet) as the root source file when creating libc.a for musl. And then slowly, we can start deleting C source files from the musl that zig bundles, and porting the code to zig instead.

Another nice optimization/research area would be to not have a separate .a file for libc, but actually export some additional functions from the same object file as the root zig source file. In this way the libc and the zig code could share implementations and shave off bloat. This would probably be helped a lot by an incremental compiler.

[andrewrk]

For Windows, someone pointed me to MidiPix which could be another source of inspiration.

[andrewrk]

#2868 has been merged and now we have a libc for Windows.

I think the proof-of-concept is done and now this issue can be split into separate issues:

• macOS: add macosx libc as a cross compilation target #2875
• FreeBSD: add FreeBSD libc as a cross compilation target #2876
• NetBSD: add NetBSD libc as a cross compilation target #2877
• OpenBSD: add OpenBSD libc as a cross compilation target #2878

Future work will be consolidating common functionality, e.g. the math functions, using one canonical implementation. See #2879 for that.

[wakaztahir]

@andrewrk Hi, I am building a small programming language, I can see that zig can compile libc, I've found the sources for libc's that zig compiles in the zig-bootstrap repository, Can you tell me which command is used, how are these libc's compiled, I am trying to implement cross compilation in my llvm programming language

https://github.com/Qinetik/chemical

The problem is that when I use lld to link a single object file that I created from llvm::Module, it gives error for unresolved printf symbols, Now I need to link against standard library using -lc but that also gives error that library not found, In this case this is a problem because I don't want my compiler to fail because lld can't find standard lib, the right way probably is to just compile libc from sources and link against it