GHC does not build with musl (ZuriHac 2021 edition)

[nh2]

In this issue I'm writing up 3 problems that broke GHC on musl (e.g. pkgsMusl.ghc, and all of static-haskell-nix on current nixpkgs) during the last half year.

We started investigating these at ZuriHac, but didn't get to the bottom; with some more weekends sunk into it, I've now figured them out.

The key problems:

1. GHC bootstrap binaries segfault on musl (link)
2. check_match: Assertion `version->filename == NULL error due to mismatching versions of ncurses in the GHC bindists (link)
3. ghc-8.6.5-binary.nix compilation fails with recompile with -fPIE (explained in this issue)

The challenging part about these problems are that they (and some fixes) were introduced in roughly the same time, such that one had to fix all 3 of them to actually get pkgsMusl.ghc to compile again, and making bisection much harder.

This meant that we were unable to make any real progress during ZuriHac 2021, and a lot more contributor free time had to be spent to figure it out.

lobby time:

An always-green Hydra that builds pkgsMusl.ghc would have saved multiple workdays of multiple people here.

---

If you are interested in these topics, here is the table of contents of problems plus their fixes, with links to the detail investigations:

1. GHC bootstrap binaries segfault on musl (GHC-8.6.5 bootstrap binary installation segfault on musl #118731)
   • I bisected this (writeup here to a musl change that suggests that our current way to use glibc bindists from GHC HQ, and patch musl in so that there are 2 different libcs used in the same program, only worked out of luck so far.
   • I plan to fix it by using the newer musl/Alpine based bindists from GHC HQ that exist for GHC > 8.10.
2. check_match: Assertion version->filename == NULLerror due to mismatching versions ofncurses` in the GHC bindists (pkgsMusl.haskell.compiler.ghc883 is broken (bootstrap with ghc865-binary fails) #85924)
   • Introduced by nixpkgs using a newer ncurses; fixed by Fix GHC bootstrap in pkgsMusl and include patch for binutils/16177 #103183. Explained in there.
3. ghc-8.6.5-binary.nix compilation fails with recompile with -fPIE
   • A new issue introduced when nixpkgs started to enable "pie" hardening for musl only. Works fine for C, but GHC cannot yet emit outputs that are ready for pie hardening.
   • Explained in detail in this issue.
   • Conflated by the fact that the nixpkgs manual doesn't make clear that pie hardening is disabled by default for glibc, because, the heading structure of that manual section is poor, making it easy to read it as part of the enabled-by-default hardening options.
   • I plan to fix it by disabling pie hardening for Haskell builds, but found is already done in master by 830ef64 via PR Haskell updates #123682.

---

This issue can be closed when all above problems are addressed on nixpkgs master.

---

CC the people involved in these topics: @expipiplus1 @bgamari @nomeata @domenkozar @dtzWill @lunaris @NorfairKing @maralorn @sternenseemann

Special thanks to @aveltras and tristanC who helped me debug a lot during ZuriHac.

[nh2]

Writeup of recompile with -fPIE issue

**ghc-8.6.5-binary.nix compilation fails with ** (explained in this issue)

I chose to bisect this on the nixpkgs repo. Build commands I considered:

• turned out to build too much because it unnecessarily builds sphinx and perl build-time-only dependencies with musl:

  NIX_PATH=nixpkgs=. nix-build -A pkgsMusl.ghc

• faster:

  NIX_PATH=nixpkgs=. nix-build --no-link --expr 'let pkgs = import <nixpkgs> {}; in pkgs.pkgsMusl.ghc.override { python3 = pkgs.python3; sphinx = pkgs.python3Packages.sphinx; perl = pkgs.perl; }'

• even faster; I used this for the bisection:

  NIX_PATH=nixpkgs=. nix-build --no-link --expr 'let pkgs = import <nixpkgs> {}; in pkgs.pkgsMusl.ghc.override { python3 = pkgs.python3; sphinx = pkgs.python3Packages.sphinx; perl = pkgs.perl; ghcFlavour = "quickest"; }'

Click to expand my bisection notes:

e9148dc1c30 nixos-21.05     `Segmentation fault` in 12:45.73 total on nixos-21.05
3a8d7958a61 nixos-unstable  segfault
b83e3e2                     ghc-8.6.5-binary.drv compilation fails with `recompile with -fPIE`
6953332fb7b nixos-20.09     ghc-8.8.4.drv failes with `check_match: Assertion `version->filename == NULL || ! _dl_name_match_p (version->filename, map)' failed!`
15b3bc33064 ghc865-bin fix  succeeds building ghc-8.10.2 with python overrides; without, it fails building spinx by Cython build failure. Success is 46:08.50 total
            from https://github.com/NixOS/nixpkgs/issues/85924


git bisect good 15b3bc33064
git bisect bad e9148dc1c30
# 14 steps
b1e42ebb                    ghc-8.6.5-binary.drv compilation fails with `recompile with -fPIE`
# Deciding to bisect first ghe 8.6.5  `recompile with -fPIE` error.
# Note this is in `running install tests`, that is, the `installCheckPhase` of ghc, that tests if the built ghc865Binary can build Haskell executables.
d9b2140                     succeeds building ghc-8.10.3; 55:53.84 total
e5d4513                     ghc-8.6.5-binary.drv compilation fails with `recompile with -fPIE`
606135b                     succeeds building ghc-8.10.3; 53:47.15 total
5d88c1c                     ghc-8.6.5-binary.drv compilation fails with `recompile with -fPIE`
23db192                     ghc-8.6.5-binary.drv compilation fails with `recompile with -fPIE`
8e7c3b9                     ghc-8.6.5-binary.drv compilation fails with `recompile with -fPIE`
b19b6cb                     ghc-8.6.5-binary.drv compilation fails with `recompile with -fPIE`
8ca7489                     succeeds building ghc-8.10.3; already built
4d69730                     ghc-8.6.5-binary.drv compilation fails with `recompile with -fPIE`
4439c8f                     ghc-8.6.5-binary.drv compilation fails with `recompile with -fPIE`
09c2f74                     succeeds building ghc-8.10.3; 18:53.97
a6840c5                     ghc-8.6.5-binary.drv compilation fails with `recompile with -fPIE`
6245758                     succeeds building ghc-8.10.3; already built
4e9dc46                     OTHER ERROR: fails with check_match: Assertion `version->filename == NULL || ! _dl_name_match_p (version->filename, map)' failed! (supposed fixed by start commit `15b3bc33064`)
                            The `good` commit is not an ancestor of this -- maybe git picked this to test a merge?
340b678                     succeeds building ghc-8.10.3; already built

The first bad commit could be any of:
4e9dc46dea0ef8cf15c567fa863796bb23099d0b
a6840c55c2e26075adae7a4a2ef4fa947f1d01b5


commit 4e9dc46dea0ef8cf15c567fa863796bb23099d0b (HEAD)
Author: Nathan Hawkins <[email protected]>
Date:   Sat Oct 24 22:34:50 2020 +0000

    stdenv: Fix hardening default for pkgsMusl to reenable -pie
    
    defaultHardeningFlags is set to enable pie for Musl, but is not
    actually used because the default is never put into
    NIX_HARDENING_ENABLE. That still works for cases other than Musl
    only because NIX_HARDENING_ENABLE is defaulted in the binutils and
    cc-wrapper setup-hook.sh scripts.

commit a6840c55c2e26075adae7a4a2ef4fa947f1d01b5 (refs/bisect/bad)
Merge: 340b6780b85 4e9dc46dea0
Author: Guillaume Girol <[email protected]>
Date:   Sat Jan 30 15:19:12 2021 +0000

    Merge pull request #101606 from utsl42/master
    
    Fix hardening default for pkgsMusl to reenable -pie

Click to expand git bisect log:

# git bisect log                                                                              :(
git bisect start
# good: [15b3bc33064eeb0cba743cad585c829b6694669c] ghc865Binary: Use binary distribution which links against ncurses6 for x86_64-linux
git bisect good 15b3bc33064eeb0cba743cad585c829b6694669c
# bad: [e9148dc1c30e02aae80cc52f68ceb37b772066f3] gnuradio3_8: 3.8.3.0 -> 3.8.3.1
git bisect bad e9148dc1c30e02aae80cc52f68ceb37b772066f3
# bad: [b1e42ebb1fe07fa47af0b9fa7d0ba5afbf55b437] noto-fonts: remove unused input
git bisect bad b1e42ebb1fe07fa47af0b9fa7d0ba5afbf55b437
# good: [d9b21406182d1c41fcb8a89fa506032667c18ba9] Merge pull request #90416 from r-ryantm/auto-update/openapi-generator-cli
git bisect good d9b21406182d1c41fcb8a89fa506032667c18ba9
# bad: [e5d4513a5166f0dfd57ca0737b7d15e649d6d47d] Merge pull request #112529 from evenbrenden/retext-remove-overrides
git bisect bad e5d4513a5166f0dfd57ca0737b7d15e649d6d47d
# good: [606135bfd85c9d52b25188f7c6c7875ebd677eb4] Merge pull request #111141 from sternenseemann/fuzzel-1.5.0
git bisect good 606135bfd85c9d52b25188f7c6c7875ebd677eb4
# bad: [5d88c1c02ec0e5fd741c40226ad65ab07705d94a] Merge pull request #111820 from r-ryantm/auto-update/react-native-debugger
git bisect bad 5d88c1c02ec0e5fd741c40226ad65ab07705d94a
# bad: [23db19263ffe4002e3eeea85196e461c28c5bee7] Merge pull request #111561 from markuskowa/fix-maint
git bisect bad 23db19263ffe4002e3eeea85196e461c28c5bee7
# bad: [8e7c3b94e292d8d340d8f15ceb5932223b2d99d5] Merge pull request #111392 from r-ryantm/auto-update/python3.7-flask-paginate
git bisect bad 8e7c3b94e292d8d340d8f15ceb5932223b2d99d5
# bad: [b19b6cb1b62ebcfcf71dfefc427de9271badbb29] Merge pull request #111341 from r-ryantm/auto-update/nextdns
git bisect bad b19b6cb1b62ebcfcf71dfefc427de9271badbb29
# good: [8ca74894e3744fa229e04bcd645c4fd8a3ea7193] Merge pull request #111267 from r-ryantm/auto-update/jgmenu
git bisect good 8ca74894e3744fa229e04bcd645c4fd8a3ea7193
# bad: [4d69730375349474d0e09574cae09daaf0009c85] Merge pull request #111263 from r-ryantm/auto-update/gpxsee
git bisect bad 4d69730375349474d0e09574cae09daaf0009c85
# bad: [4439c8f0fc1831fc69c7bd5a9ca28b63f14c8c29] Merge pull request #111309 from r-ryantm/auto-update/libsolv
git bisect bad 4439c8f0fc1831fc69c7bd5a9ca28b63f14c8c29
# good: [09c2f74a31b75691dbcaaca2619b7387b0135358] Merge pull request #111244 from marsam/update-docker-slim
git bisect good 09c2f74a31b75691dbcaaca2619b7387b0135358
# bad: [a6840c55c2e26075adae7a4a2ef4fa947f1d01b5] Merge pull request #101606 from utsl42/master
git bisect bad a6840c55c2e26075adae7a4a2ef4fa947f1d01b5
# good: [6245758fd522c9dd14296a06f7a3ac25236e9222] Merge pull request #111257 from r-ryantm/auto-update/haproxy
git bisect good 6245758fd522c9dd14296a06f7a3ac25236e9222
# skip: [4e9dc46dea0ef8cf15c567fa863796bb23099d0b] stdenv: Fix hardening default for pkgsMusl to reenable -pie
git bisect skip 4e9dc46dea0ef8cf15c567fa863796bb23099d0b
# good: [340b6780b851bfb93892c8c1819a59d7f1c7ca1a] hepmc3: 3.2.2 -> 3.2.3
git bisect good 340b6780b851bfb93892c8c1819a59d7f1c7ca1a
# only skipped commits left to test
# possible first bad commit: [a6840c55c2e26075adae7a4a2ef4fa947f1d01b5] Merge pull request #101606 from utsl42/master
# possible first bad commit: [4e9dc46dea0ef8cf15c567fa863796bb23099d0b] stdenv: Fix hardening default for pkgsMusl to reenable -pie

Conclusion of the bisection

I believe that 4e9dc46 stdenv: Fix hardening default for pkgsMusl to reenable -pie is the real culprit, but that commit itself still fails with Assertion version->filename == NULL... instead of the recompile with -fPIE error that it introduces, because at that commit itself, the fix for the Assertion error was not merged yet.

This is why we see the merge commit, which brings both together, as the culprit

Why do we get the error?

It's the installCheck of ghc865Binary that fails linking a hello-world Haskell executable.
Probably the linker is invoked with -pie now, but the Haskell .o/.a files created by GHC are not compiled with -fPIE, so they are not ready for -pie linking.

Unclear so far:

Why does it fail only with musl, and not also for normal glibc builds? Does GHC use -fPIE with glibc?

A suggestion is that only musl builds enable -pie by default: db0abe9

# XXX: Not actually musl-speciic but since only musl enables pie by default,
#      limit rebuilds by only disabling pie w/musl
  ++ stdenv.lib.optional stdenv.hostPlatform.isMusl "pie";

However, I have not found where in the code that should be specific to musl; according to https://nixos.org/manual/nixpkgs/stable/#pie, "pie" hardening is enabled by default in general.

Update:

I think I found it: https://github.com/NixOS/nixpkgs/pull/49704/files

nixpkgs/pkgs/stdenv/generic/make-derivation.nix

Lines 96 to 98 in 47a634a

	defaultHardeningFlags = if stdenv.targetPlatform.isMusl
	then supportedHardeningFlags
	else lib.remove "pie" supportedHardeningFlags;

This enables all hardening flags for isMusl, and disables "pie" for glibc.

Some side notes

• Fix hardening default for pkgsMusl to reenable -pie #101606 - "Fix hardening default for pkgsMusl to reenable -pie"
  Also interesting from there:
  • cross: regression with armv7 pkgsStatic (error: C compiler cannot create executables) #114953 - "cross: regression with armv7 pkgsStatic (error: C compiler cannot create executables)"
    • stdenv: Fix regression on ARM+static when enabling hardening #115363 - "stdenv: Fix regression on ARM+static when enabling hardening"

      native aarch64 static also produces segfaulting dynamically linked binaries
      It seems that for native x86_64-linux, static builds are fine though.

Finding the fix

With this patch:

--- a/pkgs/development/compilers/ghc/8.6.5-binary.nix
+++ b/pkgs/development/compilers/ghc/8.6.5-binary.nix
@@ -150,6 +150,8 @@ stdenv.mkDerivation rec {
     done
   '';
 
+  hardeningDisable = lib.optional stdenv.targetPlatform.isMusl "pie";
+
   doInstallCheck = true;
   installCheckPhase = ''
     unset ${libEnvVar}

The installCheck of ghc865Binary passes. However, anything it builds, e.g. happy on the way to a musl-built ghc, continues to have recompile with -fPIE errors.

We must disable "pie" hardening for all Haskell packages with musl.

This commit (is in master, not 21.05) is promising:

• 830ef64 - "haskell-generic-builder: disable static PIE"

Indeed picking this commit on top of my patch above makes happy compile.

---

From here I could continue working on the segfault issue: #118731

Some more side nodes I made during the investigation:

• -pie and -fPIE are explained in https://nixos.org/manual/nixpkgs/stable/#pie.
• GHC does not compile on pkgsMusl after the ld.gold change #84670 - "GHC does not compile on pkgsMusl after the ld.gold change"
• 830ef64 - "haskell-generic-builder: disable static PIE"
  • https://gitlab.haskell.org/ghc/ghc/-/issues/19580 - "Support creating static PIE"
• musl: Trivial binaries built with musl will fault on nixos #94228 (comment) - "musl: Trivial binaries built with musl will fault on nixos"

  This also affects non static builds.
  The most obvious way to compile something with musl, i. e. musl-gcc test.c will result in a binary which segfaults, because it has the wrong interpreter set.

• @nomeata did a similar bisection for ocaml, finding the same commit:
  pkgsMusl.ocamlPackages.ocaml: -r and -pie may not be used together #124476 (comment) - "pkgsMusl.ocamlPackages.ocaml: -r and -pie may not be used together"
• 6ebb2c3 - "ghc: disable pie, for now limit to musl since not default yet elsewhere"
  Adds hardeningDisable = ... ++ stdenv.lib.optional stdenv.targetPlatform.isMusl "pie";
  • in PR glibc, ghc: disable PIE hardening when enabled by default (musl for now :)) #50426 - "glibc, ghc: disable PIE hardening when enabled by default (musl for now :))"
    • linked PR: make-derivation: enable pie hardening with musl #49704 - "make-derivation: enable pie hardening with musl"
  • However, these hardeningDisables are for the nix-compiled GHCs, not for ghc865Binary.

CC contributors

• from the fix in master: @rnhmjoj
• from the enabling of pie hardening of musl builds: @utsl42 via Fix hardening default for pkgsMusl to reenable -pie #101606

[sternenseemann]

An always-green Hydra that builds pkgsMusl.ghc would have saved multiple workdays of multiple people here.

Feel free to add that to pkgs/top-level/release-haskell.nix as soon as its working again. We already have some CI for pkgsStatic.haskellPackages in there (with a dedicated aggregate job set).

[nh2]

PR at #129289.

Feel free to add that to pkgs/top-level/release-haskell.nix as soon as its working again. We already have some CI for pkgsStatic.haskellPackages in there (with a dedicated aggregate job set).

@sternenseemann I tried this but I haven't figured out how to exclude Darwin in there, will need some review suggestions how to do it right :)

[nh2]

I merged PR #129289 into haskell-updates that fixes this.

Close this issue when master has those changes.

[cdepillabout]

#129289 has now been merged into master, so I will go ahead and close this issue.