Allow users to choose libc version

[bramtayl]

even though it should be, in both glibc and musl. See util-linux/util-linux#1002 (comment) for some ideas about what might be going on.

[staticfloat]

We build against an exceedingly old version of glibc for compatibility reasons; O_PATH was added in glibc 2.14, but on x86_64 we build against glibc 2.12.

Luckily, we include recent kernel headers, so you can go through and anytime in the source you see #include <fcntl.h> you can replace it with #include <linux/fcntl.h> and it will use the Linux header version instead.

This isn't really something we should ask upstream to change; it's a peculiarity of how we're building things in BB, and it's non-portable; you shouldn't do this on FreeBSD and MacOS, for instance. You can build up a patch and conditionally apply it within the BB environment. :)

[bramtayl]

Oh I see. Jack2 is failing to build for me too; it doesn't seem like PRIu64 is working, which was added in C99. Could it be for similar reasons?

[bramtayl]

In any case, I think it would be nice if this were documented.

[giordano]

In any case, I think it would be nice if this were documented.

Like in https://juliapackaging.github.io/BinaryBuilder.jl/dev/tricksy_gotchas/ and https://github.com/JuliaPackaging/Yggdrasil/blob/f12d86ad889cb9880dff9c4d25bd49c9fbdd6d24/RootFS.md#glibc-versions?

[bramtayl]

Oh I hadn't seen those. But unless I'm missing something I don't see anything about how to replace with the linux headers?

[staticfloat]

@bramtayl could you submit a PR that adds a note to the tricksy_gotchas section that notes relying on newer Glibc features can often be fixed by including linux kernel headers rather than glibc headers, which is usually as easy as replacing headers such as #include <fcntl.h> with #include <linux/fcntl.h>?

[bramtayl]

Ok, will do. Sidenote I tried doing it in my example (util-linux) and C threw a bunch of errors about structs being defined more than once. I think I'll have to just add the line defining O_PATH.

[staticfloat]

Those errors are because other files are still including fcntl.h instead of linux/fcntl.h. You need to change all of them.

[bramtayl]

I wonder if it might be nicer to have a utility function to do it for users? I had a sed script written up; we could potentially build a binary of sed (or just use julia's file functions).

[staticfloat]

This is very much a case-by-case fix; It will require a user going through and changing all of the ones that need to be changed. I think a tool to fix this would need to:

(a) find all references to glibc headers within all source files in the current package.
(b) find or guess which header is causing the problem.
(c) start replacing headers in source files one by one until the whole thing compiles.

It doesn't seem easy to me to do this automatically.

[bramtayl]

Out of curiousity, what would be the problem with just using a newer version of glibc?

[giordano]

Users with an old glibc wouldn't be able to use our binaries

[bramtayl]

@staticfloat I ran

grep --recursive --files-with-matches '#include <fcntl.h>' . | xargs sed -i 's/#include <fcntl.h>/#include <linux\/fcntl.h>/'
grep --recursive --files-with-matches '# include <fcntl.h>' . | xargs sed -i 's/# include <fcntl.h>/# include <linux\/fcntl.h>/'

so I'm pretty sure that all of the fcntl.h header files have been replaced. I'm still getting the redefinition errors listed below.

In file included from /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/include/fcntl.h:34:0,
                 from ./include/c.h:278,
                 from ./include/canonicalize.h:15,
                 from libmount/src/cache.c:33:
/opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/include/bits/fcntl.h:165:8: error: redefinition of ‘struct flock’
 struct flock
        ^
In file included from /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/include/asm/fcntl.h:1:0,
                 from /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/include/linux/fcntl.h:4,
                 from libmount/src/cache.c:30:
/opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/include/asm-generic/fcntl.h:195:8: note: originally defined here
 struct flock {
        ^
In file included from /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/include/fcntl.h:34:0,
                 from ./include/c.h:278,
                 from ./include/canonicalize.h:15,
                 from libmount/src/cache.c:33:
/opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/include/bits/fcntl.h:180:8: error: redefinition of ‘struct flock64’
 struct flock64
        ^
In file included from /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/include/asm/fcntl.h:1:0,
                 from /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/include/linux/fcntl.h:4,
                 from libmount/src/cache.c:30:
/opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/include/asm-generic/fcntl.h:210:8: note: originally defined here
 struct flock64 {
        ^
/opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/include/bits/fcntl.h:194:5: error: expected identifier before numeric constant
     F_OWNER_TID = 0,  /* Kernel thread.  */
     ^
In file included from /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/include/fcntl.h:34:0,
                 from ./include/c.h:278,
                 from ./include/canonicalize.h:15,
                 from libmount/src/cache.c:33:
/opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/include/bits/fcntl.h:201:8: error: redefinition of ‘struct f_owner_ex’
 struct f_owner_ex
        ^
In file included from /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/include/asm/fcntl.h:1:0,
                 from /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/include/linux/fcntl.h:4,
                 from libmount/src/cache.c:30:
/opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/include/asm-generic/fcntl.h:155:8: note: originally defined here
 struct f_owner_ex {
        ^

Do you have ideas of what to try next? Sorry I'm so hopeless at this.

[staticfloat]

No worries at all! This is how you level up in dealing with horrible C compile problems. :)

So your error message is actually telling you precisely where the problems are. Let's peel apart the first couplet:

In file included from /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/include/fcntl.h:34:0,
                 from ./include/c.h:278,
                 from ./include/canonicalize.h:15,
                 from libmount/src/cache.c:33:
/opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/include/bits/fcntl.h:165:8: error: redefinition of ‘struct flock’
 struct flock
        ^
In file included from /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/include/asm/fcntl.h:1:0,
                 from /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/include/linux/fcntl.h:4,
                 from libmount/src/cache.c:30:
/opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/include/asm-generic/fcntl.h:195:8: note: originally defined here
 struct flock {
        ^

This says "Hey, there are two flock structures being defined! There's one defined here, and one defined here". In the "here"'s that it is complaining about it gives you the "backtrace", as it were, of file includes. So you can see three is one that comes from usr/include/fcntl.h, and one that comes from usr/include/asm/fcntl.h, which is included via usr/include/linux/fcntl.h. Because we are trying to use everything from linux/fcntl.h, this means that the first one is coming from an fcntl.h file that we would rather not include. Looking at the trail of that one, we see it comes from the include chain cache.c:33 -> canonicalize.h:15 -> c.h:278 -> fcntl.h. So my bet is that on line 278 of include/c.h in the source code, there is an #include that you need to change. :)

[bramtayl]

I've making some progress, but it looks like there's not a neat correspondence between the new headers and the old headers. I don't see sys/file.h or stdlib.h. I'm ready to give up and say you can't build util-linux using such old versions of glibc. I'm up to

grep --recursive --files-with-matches '#include <fcntl.h>' . | xargs sed -i 's/#include <fcntl.h>/#include <linux\/fcntl.h>/'
grep --recursive --files-with-matches '# include <fcntl.h>' . | xargs sed -i 's/# include <fcntl.h>/# include <linux\/fcntl.h>/'
grep --recursive --files-with-matches '#include <sys\/file.h>' . | xargs sed -i 's/#include <sys\/file.h>//'
grep --recursive --files-with-matches '#include <time.h>' . | xargs sed -i 's/#include <time.h>/#include <linux\/time.h>/'
grep --recursive --files-with-matches '#include <stdlib.h>' . | xargs sed -i 's/#include <stdlib.h>//'
grep --recursive --files-with-matches '# include <stdlib.h>' . | xargs sed -i 's/# include <stdlib.h>//'

[staticfloat]

@bramtayl can you upload your progress somewhere so I can take a look?

[bramtayl]

I've updated JuliaPackaging/Yggdrasil#1367 with my progress so far

[bramtayl]

I'm running into similar issues building systemd, and using an older version doesn't seem like it will work this time. It seems like their must be an easy solution. Maybe there's a way to add compatibility packages for newer versions of libc or something?

[bramtayl]

Users with an old glibc wouldn't be able to use our binaries

@giordano can you give more detail here? It seems to me that glibc is mostly backwards compatible, and also, it shouldn't matter because the binaries are shipped already compiled? In any case, I think there should be an option to opt into a newer version of glibc, because I'm 95% certain systemd can't be built on binarybuilder as is.

[staticfloat]

glibc is backwards compatible in that if you build against glibc 2.12, it will run on glibc 2.14. It is not forwards compatible, in that if you build on glibc 2.14 it may not run on glibc 2.12. To see examples of this, just search for error messages like version GLIBC_2.14 not found (required by xxx). These issues are quite common on the internet when downloading binaries that haven't been built against an explicitly old version of glibc.

Maybe there's a way to add compatibility packages for newer versions of libc or something?

We may actually be able to just upgrade the glibc that we require. Right now we are very, very compatible; glibc 2.12 allows us to support CentOS 6, which is EOL'ing quite soon (full updates stopped in 2017, maintenance updates stop at the end of November, 2020). CentOS 7 bumps glibc up to 2.17 and is supported through 2024, and that's the oldest distro I can find that's currently supported and that we might possibly care about.

Do you know if glibc 2.17 would be enough to build systemd?

[bramtayl]

Oh I see, never mind about the backwards compatibility thing. According to the README systemd requires glibc >= 2.16 so I think 2.17 should do it (still can't guarantee I'll be able to build systemd in the first place but I'll try my darnedest)

[bramtayl]

I think I can wait until the end of November too, there's no real urgency

[bramtayl]

@staticfloat guess what month it is?!!!

[bramtayl]

Just to clarify, it looks like CentOS6 has EOL'ed, so a newer libc should be ok?

[bramtayl]

Ok, so after a bit of thought, I think the best way to go about solving this would be to allow users to choose the version of libc they want. After all, we let them choose which version of gcc to use, so I don't see why this shouldn't be allowed either. I'm still struggling with Jack2 due to the libc version I think.

[giordano]

If I understand correctly, it's not that easy to "let the user choose the glibc version" as we'd need to build different compiler shards for each of them, which leads to an explosion of toolchains

[bramtayl]

Yes, I thought about that...and it doesn't help that they've already made it to glibc 2.33, leaving a lot of versions to choose from. I'm not sure what do about that, but I thought this might be a little more palatable than JuliaPackaging/Yggdrasil#2435 (which didn't seem to go anywhere), because you can blame any version incompatibilities on the user

[pepijndevos]

If we could bump glibc to 2.17 as suggested, that would also probably solve JuliaPackaging/Yggdrasil#3513

The function aligned_alloc() was added to glibc in version 2.16.
https://linux.die.net/man/3/aligned_alloc