ld.gold crashes or produces invalid executables when Musl is used as a libc #49071

albertov · 2018-10-24T18:43:29Z

Issue description

ld.gold crashes or produces invalid executables when using the musl libc. This prevents the use of musl and GHC on armv7l (where it also manifests itself) since GHC requires ld.gold on that platform to work around https://sourceware.org/bugzilla/show_bug.cgi?id=16177

This also affects lld from LLVM 4 apparently

Steps to reproduce

nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideDerivation(_: {CFLAGS="-fuse-ld=gold";})'

The above command either fails (on a bare-metal x86_64 system) or produces an executable which segfaults at start (on a x86_64 system running on VM in a laptop). Interestingly, if one passes -static_libgcc the produced executable is apparently correct inside the VM but also fails to be created on the bare-metal machine.

Technical details

$ nix-info -m
 - system: `"x86_64-linux"`
 - host os: `Linux 4.14.78, NixOS, 18.09.1007.f8d156f44fb (Jellyfish)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.1pre6377_954d1f4d`
 - channels(root): `"nixos-18.09.1007.f8d156f44fb"`
 - channels(alberto): `"nixos-18.03pre120031.7ebacd1a43, nixpkgs-18.03pre119383.eafd703a63"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

The above system is the desktop bare-metal one

The text was updated successfully, but these errors were encountered:

nh2 · 2018-10-25T11:44:01Z

Just for clarity, is this the error you get when compiling on plain x86?

NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideDerivation(_: {CFLAGS="-fuse-ld=gold";})'
these derivations will be built:
  /nix/store/c7d5hz6ynzc6gyg69w2ja8jdx11537zl-hello-2.10.drv
these paths will be fetched (0.69 MiB download, 0.69 MiB unpacked):
  /nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz
copying path '/nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz' from 'https://cache.nixos.org'...
building '/nix/store/c7d5hz6ynzc6gyg69w2ja8jdx11537zl-hello-2.10.drv'...
unpacking sources
unpacking source archive /nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz
source root is hello-2.10
setting SOURCE_DATE_EPOCH to timestamp 1416139241 of file hello-2.10/ChangeLog
patching sources
configuring
configure flags: --disable-dependency-tracking --prefix=/nix/store/wdl72bp6a41iiqv5ph37nwqbhx1sf825-hello-2.10
checking for a BSD-compatible install... /nix/store/bxwz6wdxr0lgj9c629azmxi35i68qrv2-coreutils-8.29/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /nix/store/bxwz6wdxr0lgj9c629azmxi35i68qrv2-coreutils-8.29/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... configure: error: in `/tmp/nix-build-hello-2.10.drv-0/hello-2.10':
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details
builder for '/nix/store/c7d5hz6ynzc6gyg69w2ja8jdx11537zl-hello-2.10.drv' failed with exit code 1

albertov · 2018-10-25T11:48:22Z

Exactly. If you see the config.log it should show that the linker has refused to link. Should also be able to reproduce in nix-shell.

…

On Thu, 25 Oct 2018 at 13:44, Niklas Hambüchen ***@***.***> wrote: Just for clarity, is this the error you get when compiling on plain x86? NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideDerivation(_: {CFLAGS="-fuse-ld=gold";})' these derivations will be built: /nix/store/c7d5hz6ynzc6gyg69w2ja8jdx11537zl-hello-2.10.drv these paths will be fetched (0.69 MiB download, 0.69 MiB unpacked): /nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz copying path '/nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz' from 'https://cache.nixos.org'... building '/nix/store/c7d5hz6ynzc6gyg69w2ja8jdx11537zl-hello-2.10.drv'... unpacking sources unpacking source archive /nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz source root is hello-2.10 setting SOURCE_DATE_EPOCH to timestamp 1416139241 of file hello-2.10/ChangeLog patching sources configuring configure flags: --disable-dependency-tracking --prefix=/nix/store/wdl72bp6a41iiqv5ph37nwqbhx1sf825-hello-2.10 checking for a BSD-compatible install... /nix/store/bxwz6wdxr0lgj9c629azmxi35i68qrv2-coreutils-8.29/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /nix/store/bxwz6wdxr0lgj9c629azmxi35i68qrv2-coreutils-8.29/bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking for gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... configure: error: in `/tmp/nix-build-hello-2.10.drv-0/hello-2.10': configure: error: cannot run C compiled programs. If you meant to cross compile, use `--host'. See `config.log' for more details builder for '/nix/store/c7d5hz6ynzc6gyg69w2ja8jdx11537zl-hello-2.10.drv' failed with exit code 1 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#49071 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA00CyJD-lmz9ZrnbexYPzfSdMIAdTLNks5uoaQJgaJpZM4X4kDl> .

nh2 · 2018-10-25T11:51:14Z

I see, in config.log:

configure:4154: checking whether we are cross compiling
configure:4162: gcc -o conftest -fuse-ld=gold   conftest.c  >&5
configure:4166: $? = 0
configure:4173: ./conftest
./configure: line 4175: 14393 Segmentation fault      ./conftest$ac_cv_exeext
configure:4177: $? = 139
configure:4184: error: in `/tmp/nix-build-hello-2.10.drv-0/hello-2.10':
configure:4186: error: cannot run C compiled programs.

albertov · 2018-10-25T12:07:53Z

This is the same case that happens in my VM I think. If you try to compile a simple C program inside the nix-shell of hello with -static-libgcc then the produced executable should work, for some reason. Its really curious that on my bare metal machine ld.gold just bails. I’ve seen the gcc, binutils and musl derivations are the same on both machines yet they behave differently. Are you running it inside a VM?

…

On Thu, 25 Oct 2018 at 13:51, Niklas Hambüchen ***@***.***> wrote: I see, in config.log: configure:4154: checking whether we are cross compiling configure:4162: gcc -o conftest -fuse-ld=gold conftest.c >&5 configure:4166: $? = 0 configure:4173: ./conftest ./configure: line 4175: 14393 Segmentation fault ./conftest$ac_cv_exeext configure:4177: $? = 139 configure:4184: error: in `/tmp/nix-build-hello-2.10.drv-0/hello-2.10': configure:4186: error: cannot run C compiled programs. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#49071 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA00C7t2_w5o4zz8NUSI1WomtU60MZPAks5uoaW6gaJpZM4X4kDl> .

nh2 · 2018-10-25T13:04:26Z

Are you running it inside a VM?

No, I'm running on Nix on a standard Ubuntu 16.04 64-bit.

nh2 · 2018-10-25T13:23:31Z

Related resources:

Looking at nm output like done in that thread:

niklas@ares:/tmp/nix-build-hello-2.10.drv-0/hello-2.10/src$ ls
core  hello.c  system.h  test.c
niklas@ares:/tmp/nix-build-hello-2.10.drv-0/hello-2.10/src$ cat test.c 
int main(int argc, const char *argv[]) {
  return 0;
}
niklas@ares:/tmp/nix-build-hello-2.10.drv-0/hello-2.10/src$ gcc -o hello.bfd test.c
niklas@ares:/tmp/nix-build-hello-2.10.drv-0/hello-2.10/src$ gcc -o hello.gold -fuse-ld=gold test.c

niklas@ares ...build-hello-2.10.drv-0/hello-2.10/src % nm -D conftest-gold
0000000000402008 D __bss_start
                 w __deregister_frame_info
0000000000402008 D _edata
0000000000402070 D _end
00000000004007e2 T _fini
00000000004005b8 T _init
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
                 U __libc_start_main
                 w __register_frame_info
niklas@ares ...build-hello-2.10.drv-0/hello-2.10/src % nm -D conftest-bfd
0000000000601008 B __bss_start
0000000000601008 D _edata
0000000000601070 B _end
00000000004006e2 T _fini
00000000004004e8 T _init
                 U __libc_start_main

dtzWill · 2018-10-25T13:25:47Z

FWIW it seems to work with clang, which is curious:

NIX_PATH=nixpkgs=channel:nixos-unstable nix-build -E 'with (import <nixpkgs> {}).pkgsMusl; (pkgs.hello.override { inherit (llvmPackages_6) stdenv; }).overrideAttrs (o:{ CFLAGS="-fuse-ld=gold"; NIX_DEBUG=1;})'

As for using gold, may want to try something like: 37b2f59

Or otherwise ensuring our wrappers always know what linker to invoke. I don't think this is the problem, but something to consider for future gold usage once this is resolved :).

dtzWill · 2018-10-25T13:32:47Z

https://www.openwall.com/lists/musl/2015/05/01/3

The thread mentions building with -fPIE, which is interesting because with NIX_DEBUG=1 I'm seeing this in the log:

HARDENING: disabled flags: pie
HARDENING: Is active (not completely disabled with "all" flag)
HARDENING: enabling fortify
HARDENING: enabling stackprotector
HARDENING: enabling strictoverflow
HARDENING: enabling format
HARDENING: enabling pic

Which may be related? It's disabled on non-musl too, FWIW.

nh2 · 2018-10-25T13:53:23Z

Adding -pie to LDFLAGS makes it work:

NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideDerivation(_: {CFLAGS="-fuse-ld=gold"; LDFLAGS="-pie";})'

Adding -fPIE to CFLAGS doens't help, it seems to be removed (perhaps by the hardening)?

nh2 · 2018-10-25T14:08:15Z

Also does anybody have an idea why

NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideDerivation(_: {CFLAGS="-fuse-ld=gold -fPIE"; hardeningDisable = [ "all" ];})'

doesn't work (using CFLAGS="-fuse-ld=gold -fPIE" and hardeningDisable = [ "all" ])?

nh2 · 2018-10-25T14:15:18Z

seems to be removed (perhaps by the hardening)?

Just asked @fpletz at NixCon, he said that hardening should only add flags, not remove them; @cleverca22 said that happens here: https://github.com/NixOS/nixpkgs/blob/62feeed1a5aaf23b8c0883ae0cc17cd913cdce56/pkgs/build-support/cc-wrapper/add-hardening.sh

So not sure why -fPIE doesn't translate to -pie to the linker.

nh2 · 2018-10-25T14:24:50Z

So not sure why -fPIE doesn't translate to -pie to the linker.

With help of @cleverca22 and looking at

nixpkgs/pkgs/build-support/cc-wrapper/add-hardening.sh

Lines 47 to 52 in 62feeed

    
           pie) 
        
             if (( "${NIX_DEBUG:-0}" >= 1 )); then echo HARDENING: enabling CFlags -fPIE >&2; fi 
        
             hardeningCFlags+=('-fPIE') 
        
             if [[ ! ("$*" =~ " -shared " || "$*" =~ " -static ") ]]; then 
        
               if (( "${NIX_DEBUG:-0}" >= 1 )); then echo HARDENING: enabling LDFlags -pie >&2; fi 
        
               hardeningCFlags+=('-pie')

It seems you need to pass both -pie and -fPIE to the compiler; see man gcc:

       -pie
           Produce a position independent executable on targets that support it.
           For predictable results, you must also specify the same set of
           options used for compilation (-fpie, -fPIE, or model suboptions) when
           you specify this linker option.
...
       -fpie
       -fPIE
           These options are similar to -fpic and -fPIC, but generated position
           independent code can be only linked into executables.  Usually these
           options are used when -pie GCC option is used during linking.

           -fpie and -fPIE both define the macros "__pie__" and "__PIE__".  The
           macros have the value 1 for -fpie and 2 for -fPIE.

And indeed

NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideDerivation(_: {CFLAGS="-fuse-ld=gold -fPIE -pie";})'

works.

nh2 · 2018-10-25T14:29:56Z

I'd expect that hardeningEnable = ["pie"]; should also work, but it doesn't:

NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideDerivation(_: {CFLAGS="-fuse-ld=gold"; hardeningEnable = ["pie"]; })'

Contrary to expectations, this doesn't seem to add the pie flags.

dtzWill · 2018-10-25T16:38:19Z

It works if you use overrideAttrs instead! \o/

…

On Thu, 25 Oct 2018 07:30:03 -0700, Niklas Hambüchen ***@***.***> wrote: I'd expect that `hardeningEnable = ["pie"];` should also work, but it doesn't: ``` NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideDerivation(_: {CFLAGS="-fuse-ld=gold"; hardeningEnable = ["pie"]; })' ``` Contrary to expectations, this doesn't seem to add the pie flags. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #49071 (comment) part: text/html

nh2 · 2018-10-25T16:46:19Z

It works if you use overrideAttrs instead! \o/

Oh right, this works!

NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideAttrs(_: {CFLAGS="-fuse-ld=gold"; hardeningEnable = ["pie"]; })'

@dtzWill Should we add something to the musl bits that detects if gold is used and passes pie accordingly?

Also, I assume there should be some upstream gold bug filed somewhere? The musl thread didn't link any being filed.

albertov · 2018-10-25T17:05:46Z

The command which worked for @nh2 :

NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideDerivation(_: {CFLAGS="-fuse-ld=gold -fPIE -pie";})'

Failed in my machine :( Same error as always:

checking whether the C compiler works... no
configure: error: in `/build/hello-2.10':
configure: error: C compiler cannot create executables
See `config.log' for more details
builder for '/nix/store/g6m7i019af8p838ldzv4jlfxds40fw2g-hello-2.10.drv' failed with exit code 77
error: build of '/nix/store/g6m7i019af8p838ldzv4jlfxds40fw2g-hello-2.10.drv' failed

dtzWill · 2018-10-25T18:02:28Z

How about using overrideAttrs instead? Otherwise it didn't work for me either. Thisis probably due to the same sorts of things that make overrideAttrs generally what you should use: https://nixos.org/nixpkgs/manual/#sec-pkg-overrideAttrs (and warning at top of following blurb about overrideDerivation). Unfortunately I personally remember this is true for one of them, but never which one is which. Luckily muscle memory usually gets it right as long as I don't think about it too much haha.

…

On Thu, Oct 25, 2018, 12:05 PM Alberto Valverde ***@***.***> wrote: The command which worked for @nh2 <https://github.com/nh2> : NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideDerivation(_: {CFLAGS="-fuse-ld=gold -fPIE -pie";})' Failed in my machine :( Same error as always: checking whether the C compiler works... no configure: error: in `/build/hello-2.10': configure: error: C compiler cannot create executables See `config.log' for more details builder for '/nix/store/g6m7i019af8p838ldzv4jlfxds40fw2g-hello-2.10.drv' failed with exit code 77 error: build of '/nix/store/g6m7i019af8p838ldzv4jlfxds40fw2g-hello-2.10.drv' failed — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#49071 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAx4si-ezr7zcBqzmca4l5NjTCIlH7jiks5uoe9wgaJpZM4X4kDl> .

albertov · 2018-10-25T18:09:09Z

Didn't know about overrideAttrs (thanks for letting me know!). Anyway, I get the same error with overrideAttrs too, this is what I'm running and the output:

3$ NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideAttrs(_: {CFLAGS="-fuse-ld=gold -fPIE -pie";})'
these derivations will be built:
  /nix/store/g6m7i019af8p838ldzv4jlfxds40fw2g-hello-2.10.drv
building '/nix/store/g6m7i019af8p838ldzv4jlfxds40fw2g-hello-2.10.drv'...
unpacking sources
unpacking source archive /nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz
source root is hello-2.10
setting SOURCE_DATE_EPOCH to timestamp 1416139241 of file hello-2.10/ChangeLog
patching sources
configuring
configure flags: --disable-dependency-tracking --prefix=/nix/store/rmwms3g2v6isnswxlm1w799vb82zfp24-hello-2.10
checking for a BSD-compatible install... /nix/store/bxwz6wdxr0lgj9c629azmxi35i68qrv2-coreutils-8.29/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /nix/store/bxwz6wdxr0lgj9c629azmxi35i68qrv2-coreutils-8.29/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for gcc... gcc
checking whether the C compiler works... no
configure: error: in `/build/hello-2.10':
configure: error: C compiler cannot create executables
See `config.log' for more details
builder for '/nix/store/g6m7i019af8p838ldzv4jlfxds40fw2g-hello-2.10.drv' failed with exit code 77
error: build of '/nix/store/g6m7i019af8p838ldzv4jlfxds40fw2g-hello-2.10.drv' failed

The other command which worked for you, with hardeningEnable, also fails:

$ NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideAttrs(_: {CFLAGS="-fuse-ld=gold"; hardeningEnable = ["pie"]; })'                  
these derivations will be built:
  /nix/store/qllg7c478rnba15rzh73va3adlw88zhk-hello-2.10.drv
building '/nix/store/qllg7c478rnba15rzh73va3adlw88zhk-hello-2.10.drv'...
unpacking sources
unpacking source archive /nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz
source root is hello-2.10
setting SOURCE_DATE_EPOCH to timestamp 1416139241 of file hello-2.10/ChangeLog
patching sources
configuring
configure flags: --disable-dependency-tracking --prefix=/nix/store/2k3lazi3vkm73gm033dxnlc8svasnzid-hello-2.10
checking for a BSD-compatible install... /nix/store/bxwz6wdxr0lgj9c629azmxi35i68qrv2-coreutils-8.29/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /nix/store/bxwz6wdxr0lgj9c629azmxi35i68qrv2-coreutils-8.29/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for gcc... gcc
checking whether the C compiler works... no
configure: error: in `/build/hello-2.10':
configure: error: C compiler cannot create executables
See `config.log' for more details
builder for '/nix/store/qllg7c478rnba15rzh73va3adlw88zhk-hello-2.10.drv' failed with exit code 77
error: build of '/nix/store/qllg7c478rnba15rzh73va3adlw88zhk-hello-2.10.drv' failed

My computer must be cursed :(

nh2 · 2018-10-25T23:11:39Z

Is this now also a plain x86_64 machine or something else?

albertov · 2018-10-26T00:41:57Z

Yes, it's a plain x86_64 machine running Nixos on bare hardware.

albertov · 2018-10-26T09:40:53Z

For the record, inside a nixos VM in a laptop, passing hardeningEnable=["pie"] makes it build correctly and the executable works. Without hardeningEnable (the original command from the description) the executable segfaults.

I can't understand the discrepancies between machines, both are x86_64 systems. The laptop is a macbook pro while the desktop has this processor according to /proc/cpuinfo:

processor       : 15
vendor_id       : GenuineIntel
cpu family      : 6
model           : 79
model name      : Intel(R) Core(TM) i7-6900K CPU @ 3.20GHz
stepping        : 1
microcode       : 0xb00001c
cpu MHz         : 3499.257
cache size      : 20480 KB
physical id     : 0
siblings        : 16
core id         : 7
cpu cores       : 8
apicid          : 15
initial apicid  : 15
fpu             : yes
fpu_exception   : yes
cpuid level     : 20
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 6398.64
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

I'll try to run it on other machines today to see if I can spot a pattern

albertov · 2018-10-26T09:44:32Z

This is /proc/cpuinfo from the laptop:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 70
model name	: Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz
stepping	: 1
microcode	: 0x19
cpu MHz		: 2194.268
cache size	: 6144 KB
physical id	: 6
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc cpuid aperfmperf pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault epb invpcid_single pti tpr_shadow vnmi ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid xsaveopt dtherm ida arat pln pts
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips	: 4388.53
clflush size	: 64
cache_alignment	: 64
address sizes	: 42 bits physical, 48 bits virtual
power management:

nh2 · 2018-10-26T21:36:15Z

That is pretty odd. Perhaps comparing the full config.log output gives a hint (ideally also with NIX_DEBUG=1 set)?

albertov · 2018-10-28T12:17:20Z

@nh2 I will try that and post the diff if different. I meanwhile tried to compile using clang 6 with:

NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'with (import <nixpkgs> {}).pkgsMusl; (pkgs.hello.override { inherit (llvmPackages_6) stdenv; }).overrideAttrs (o:{ CFLAGS="-fuse-ld=gold"; NIX_DEBUG=1;})'

but llvm fails to build since the gold tests fail with a very similar error:

<....snip....>
Command Output (stderr):
--
/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: fatal error: /build/llvm/build/test/tools/gold/X86/Output/visibility.ll.tmp.so: No error information

--

********************
Testing: 0 .. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
Testing Time: 165.74s
********************
Failing Tests (34):
    LLVM :: tools/gold/PowerPC/mtriple.ll
    LLVM :: tools/gold/X86/asm_undefined.ll
    LLVM :: tools/gold/X86/asm_undefined2.ll
    LLVM :: tools/gold/X86/bcsection.ll
    LLVM :: tools/gold/X86/cache.ll
    LLVM :: tools/gold/X86/comdat.ll
    LLVM :: tools/gold/X86/common_thinlto.ll
    LLVM :: tools/gold/X86/emit-llvm.ll
    LLVM :: tools/gold/X86/global_with_section.ll
    LLVM :: tools/gold/X86/mixed_lto.ll
    LLVM :: tools/gold/X86/module_asm.ll
    LLVM :: tools/gold/X86/multiple-data.s
    LLVM :: tools/gold/X86/multiple-sections.ll
    LLVM :: tools/gold/X86/opt-level.ll
    LLVM :: tools/gold/X86/parallel.ll
    LLVM :: tools/gold/X86/pr19901.ll
    LLVM :: tools/gold/X86/pr19901_thinlto.ll
    LLVM :: tools/gold/X86/pr25907.ll
    LLVM :: tools/gold/X86/relax-relocs.ll
    LLVM :: tools/gold/X86/relocatable.ll
    LLVM :: tools/gold/X86/relocation-model-pic.ll
    LLVM :: tools/gold/X86/slp-vectorize.ll
    LLVM :: tools/gold/X86/strip_names.ll
1 warning(s) in tests.

    LLVM :: tools/gold/X86/thinlto.ll
    LLVM :: tools/gold/X86/thinlto_afdo.ll
    LLVM :: tools/gold/X86/thinlto_archive.ll
    LLVM :: tools/gold/X86/thinlto_funcimport.ll
    LLVM :: tools/gold/X86/thinlto_internalize.ll
    LLVM :: tools/gold/X86/thinlto_linkonceresolution.ll
    LLVM :: tools/gold/X86/thinlto_weak_library.ll
    LLVM :: tools/gold/X86/thinlto_weak_resolution.ll
    LLVM :: tools/gold/X86/type-merge2.ll
    LLVM :: tools/gold/X86/vectorize.ll
    LLVM :: tools/gold/X86/visibility.ll

  Expected Passes    : 22394
  Expected Failures  : 140
  Unsupported Tests  : 721
  Unexpected Failures: 34
make[3]: *** [CMakeFiles/check-all.dir/build.make:58: CMakeFiles/check-all] Error 1
make[3]: Leaving directory '/build/llvm/build'
make[2]: *** [CMakeFiles/Makefile2:356: CMakeFiles/check-all.dir/all] Error 2
make[2]: Leaving directory '/build/llvm/build'
make[1]: *** [CMakeFiles/Makefile2:363: CMakeFiles/check-all.dir/rule] Error 2
make[1]: Leaving directory '/build/llvm/build'
make: *** [Makefile:251: check-all] Error 2
builder for '/nix/store/qsgcngh2m66k188d1nmqsmdy8z3j312i-llvm-6.0.1.drv' failed with exit code 2

albertov · 2018-10-28T21:06:59Z

I've compared the config.logs of the following command on both machines:

NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build -E 'let pkgs = (import <nixpkgs> {}).pkgsMusl; in pkgs.hello.overrideAttrs(_: {NIX_DEBUG="1"; CFLAGS="-fuse-ld=gold -Wl,--debug=all"; hardeningEnable = ["pie"]; postInstall="exit -1";})' -K

(the postIntall is to intentionally kill the build before it finishes correctly to force the temporary build dir to be preserved in the machine where it works properly)

If I understand it correctly, the "broken" ld.gold has some problem processing libgcc.a since when both logs begin to differ significantly is where the good version starts running task Add_archive_symbols while the bad version never reached that phase.

The beginning of the diff looks like this:

--- hello-config-bad.log    2018-10-28 21:27:33.391979234 +0100
+++ hello-config-good.log 2018-10-28 21:22:29.845499594 +0100
@@ -12,9 +12,9 @@
 
 hostname = localhost
 uname -m = x86_64
-uname -r = 4.14.78
+uname -r = 4.14.76
 uname -s = Linux
-uname -v = #1-NixOS SMP Sat Oct 20 07:48:54 UTC 2018
+uname -v = #1-NixOS SMP Sat Oct 13 07:27:30 UTC 2018
 
 /usr/bin/uname -p = unknown
 /bin/uname -X     = unknown
@@ -188,7 +188,7 @@
   -plugin
   /nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/libexec/gcc/x86_64-unknown-linux-musl/7.3.0/liblto_plugin.so
   -plugin-opt=/nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/libexec/gcc/x86_64-unknown-linux-musl/7.3.0/lto-wrapper
-  -plugin-opt=-fresolution=/build/cciodIpK.res
+  -plugin-opt=-fresolution=/build/ccbNLFDA.res
   -plugin-opt=-pass-through=-lgcc
   -plugin-opt=-pass-through=-lgcc_s
   -plugin-opt=-pass-through=-lc
@@ -214,7 +214,7 @@
   -dynamic-linker
   /nix/store/35mb6imifzx9316y7nknhh6apx1k7a24-musl-1.1.19/lib/ld-musl-x86_64.so.1
   --debug=all
-  /build/cceIfKHc.o
+  /build/ccnaDnPl.o
   -rpath
   /nix/store/xsngrd6w0r2aaqc3fdzyi16a4hpndwa2-hello-2.10/lib64
   -rpath
@@ -279,14 +279,14 @@
 /nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 completed task Read_symbols /nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/lib/gcc/x86_64-unknown-linux-musl/7.3.0/crtbeginS.o (user: 0.000000 sys: 0.000000 wall: 0.000000)
 /nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 running   task Add_symbols /nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/lib/gcc/x86_64-unknown-linux-musl/7.3.0/crtbeginS.o
 /nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 completed task Add_symbols /nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/lib/gcc/x86_64-unknown-linux-musl/7.3.0/crtbeginS.o (user: 0.000000 sys: 0.000000 wall: 0.000000)
-/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 running   task Read_symbols /build/cceIfKHc.o
-/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Opened new descriptor 9 for "/build/cceIfKHc.o"
-/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Attempt to open /build/cceIfKHc.o succeeded
-/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Unlocking file "/build/cceIfKHc.o"
-/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Released descriptor 9 for "/build/cceIfKHc.o"
-/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 completed task Read_symbols /build/cceIfKHc.o (user: 0.000000 sys: 0.000000 wall: 0.000000)
-/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 running   task Add_symbols /build/cceIfKHc.o
-/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 completed task Add_symbols /build/cceIfKHc.o (user: 0.000000 sys: 0.000000 wall: 0.000000)
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 running   task Read_symbols /build/ccnaDnPl.o
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Opened new descriptor 9 for "/build/ccnaDnPl.o"
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Attempt to open /build/ccnaDnPl.o succeeded
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Unlocking file "/build/ccnaDnPl.o"
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Released descriptor 9 for "/build/ccnaDnPl.o"
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 completed task Read_symbols /build/ccnaDnPl.o (user: 0.000000 sys: 0.000000 wall: 0.000000)
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 running   task Add_symbols /build/ccnaDnPl.o
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 completed task Add_symbols /build/ccnaDnPl.o (user: 0.000000 sys: 0.000000 wall: 0.000000)
 /nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 running   task Read_symbols /nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/lib/gcc/x86_64-unknown-linux-musl/7.3.0/crtendS.o
 /nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Opened new descriptor 10 for "/nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/lib/gcc/x86_64-unknown-linux-musl/7.3.0/crtendS.o"
 /nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Attempt to open /nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/lib/gcc/x86_64-unknown-linux-musl/7.3.0/crtendS.o succeeded
@@ -388,7 +388,25677 @@
 /nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Attempt to open /nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/lib/gcc/x86_64-unknown-linux-musl/7.3.0/libgcc.a succeeded
 /nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Unlocking file "/nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/lib/gcc/x86_64-unknown-linux-musl/7.3.0/libgcc.a"
 /nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Released descriptor 11 for "/nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/lib/gcc/x86_64-unknown-linux-musl/7.3.0/libgcc.a"
-/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 completed task Read_symbols -lgcc (user: 0.000000 sys: 0.010000 wall: 0.000000)
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 completed task Read_symbols -lgcc (user: 0.000000 sys: 0.000000 wall: 0.000000)
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 running   task Add_archive_symbols /nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/lib/gcc/x86_64-unknown-linux-musl/7.3.0/libgcc.a
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 completed task Add_archive_symbols /nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/lib/gcc/x86_64-unknown-linux-musl/7.3.0/libgcc.a (user: 0.000000 sys: 0.000000 wall: 0.000000)
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold:   0 running   task Read_symbols -lgcc_s
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Attempt to open /nix/store/35mb6imifzx9316y7nknhh6apx1k7a24-musl-1.1.19/lib/libgcc_s.so failed
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Attempt to open /nix/store/35mb6imifzx9316y7nknhh6apx1k7a24-musl-1.1.19/lib/libgcc_s.a failed
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Closed descriptor 11 for "/nix/store/4k79vw3p32n2w7z11d9rh4788chq73ja-gcc-7.3.0/lib/gcc/x86_64-unknown-linux-musl/7.3.0/libgcc.a"
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Opened new descriptor 12 for "/nix/store/4dw6k7sdkw8r2b5rgka8r659451l3xx9-gcc-7.3.0-lib/lib/libgcc_s.so"
+/nix/store/8jpp32r1yqzqcgjycnvjy0hm30ng2f86-binutils-2.30/bin/ld.gold: Attempt to open /nix/store/4dw6k7sdkw8r2b5rgka8r659451l3xx9-gcc-7.3.0-lib/lib/libgcc_s.so succeeded

matthewbauer · 2018-10-31T06:30:19Z

"pie" hardening is not enabled by default. Maybe it should be?

nh2 · 2018-11-03T17:47:40Z

"pie" hardening is not enabled by default. Maybe it should be?

@matthewbauer The way I read the links is that gold for unknown reasons (likely a bug) doesn't work with musl unless pie is activated.

Perhaps it should be enabled by default for this combination (gold + musl)?

matthewbauer · 2018-11-03T18:55:48Z

Yeah sounds like a good idea. We don't know whether you will use ld.gold ahead of time, so better to just do it with musl everywhere.

Fixes NixOS#49071 On ld.gold, we produce broken executables when linking with the Musl libc. This appears to be a known bug when using ld.gold and Musl. This thread describes the workaround as enabling PIE when using ld.gold and Musl: https://www.openwall.com/lists/musl/2015/05/01/5 By default we don’t enable PIE to avoid breaking things. But in the Musl case we are breaking things by not enabling PIE. So this adds a special case for defaultHardeningFlags which keeps the pie hardening for everything. Any packages that break with PIE can add the pie flag to disableHardeningFlags array (a no-op for now on anything but Musl).

nh2 · 2018-11-04T11:54:55Z

I've filed an upstream issue for gold at https://sourceware.org/bugzilla/show_bug.cgi?id=23856

I recommend everyone involved subscribing to it.

Fixes #49071 On ld.gold, we produce broken executables when linking with the Musl libc. This appears to be a known bug when using ld.gold and Musl. This thread describes the workaround as enabling PIE when using ld.gold and Musl: https://www.openwall.com/lists/musl/2015/05/01/5 By default we don’t enable PIE to avoid breaking things. But in the Musl case we are breaking things by not enabling PIE. So this adds a special case for defaultHardeningFlags which keeps the pie hardening for everything. Any packages that break with PIE can add the pie flag to disableHardeningFlags array (a no-op for now on anything but Musl).

ld.gold doesn't play well with musl as is documented in NixOS#49071 and https://sourceware.org/bugzilla/show_bug.cgi?id=23856

ld.gold doesn't play well with musl as is documented in #49071 and https://sourceware.org/bugzilla/show_bug.cgi?id=23856

ld.gold doesn't play well with musl as is documented in NixOS#49071 and https://sourceware.org/bugzilla/show_bug.cgi?id=23856 (cherry picked from commit 4675649)

albertov changed the title ~~ld.gold crashes or produces invalid executables when Musl is used as a libc~~ ld.gold crashes or produces invalid executables when Musl is used as a libc Oct 24, 2018

albertov mentioned this issue Oct 25, 2018

Fully static Haskell executables - overview issue #43795

Open

60 tasks

nh2 mentioned this issue Nov 3, 2018

<nix/fetchurl.nix> auto-uncompresses xz files NixOS/nix#2512

Open

matthewbauer mentioned this issue Nov 3, 2018

make-derivation: enable pie hardening with musl #49704

Merged

globin closed this as completed in #49704 Nov 7, 2018

nh2 mentioned this issue Jun 11, 2019

Support musl-based Linux distributions like Alpine Linux commercialhaskell/stack#2387

Closed

utdemir mentioned this issue Apr 7, 2020

GHC does not compile on pkgsMusl after the ld.gold change #84670

Closed

adamse added a commit to adamse/nixpkgs that referenced this issue Apr 8, 2020

ghc: don't use ld.gold with musl libc (fixes NixOS#84670)

acfccf7

ld.gold doesn't play well with musl as is documented in NixOS#49071 and https://sourceware.org/bugzilla/show_bug.cgi?id=23856

peti pushed a commit that referenced this issue Apr 10, 2020

ghc: don't use ld.gold with musl libc (fixes #84670)

4675649

ld.gold doesn't play well with musl as is documented in #49071 and https://sourceware.org/bugzilla/show_bug.cgi?id=23856

michivi mentioned this issue Apr 17, 2020

fix: not use ld.gold on aarch32 input-output-hk/haskell.nix#548

Merged

bb010g mentioned this issue May 21, 2020

cabal2nix evaluation failures nh2/static-haskell-nix#73

Open

utsl42 mentioned this issue Oct 23, 2020

ld.gold crashes or produces broken executables #101490

Open

Mikolaj mentioned this issue Jun 26, 2021

Add sdl2 and sdl2-ttf to the docker images? utdemir/ghc-musl#14

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ld.gold crashes or produces invalid executables when Musl is used as a libc #49071

ld.gold crashes or produces invalid executables when Musl is used as a libc #49071

albertov commented Oct 24, 2018 •

edited

Loading

nh2 commented Oct 25, 2018

albertov commented Oct 25, 2018 via email

nh2 commented Oct 25, 2018

albertov commented Oct 25, 2018 via email

nh2 commented Oct 25, 2018

nh2 commented Oct 25, 2018

dtzWill commented Oct 25, 2018

dtzWill commented Oct 25, 2018

nh2 commented Oct 25, 2018

nh2 commented Oct 25, 2018

nh2 commented Oct 25, 2018

nh2 commented Oct 25, 2018

nh2 commented Oct 25, 2018

dtzWill commented Oct 25, 2018 via email

nh2 commented Oct 25, 2018

albertov commented Oct 25, 2018

dtzWill commented Oct 25, 2018 via email

albertov commented Oct 25, 2018

nh2 commented Oct 25, 2018

albertov commented Oct 26, 2018

albertov commented Oct 26, 2018 •

edited

Loading

albertov commented Oct 26, 2018

nh2 commented Oct 26, 2018

albertov commented Oct 28, 2018 •

edited

Loading

albertov commented Oct 28, 2018

matthewbauer commented Oct 31, 2018 •

edited

Loading

nh2 commented Nov 3, 2018

matthewbauer commented Nov 3, 2018

nh2 commented Nov 4, 2018 •

edited

Loading

ld.gold crashes or produces invalid executables when Musl is used as a libc #49071

ld.gold crashes or produces invalid executables when Musl is used as a libc #49071

Comments

albertov commented Oct 24, 2018 • edited Loading

Issue description

Steps to reproduce

Technical details

nh2 commented Oct 25, 2018

albertov commented Oct 25, 2018 via email

nh2 commented Oct 25, 2018

albertov commented Oct 25, 2018 via email

nh2 commented Oct 25, 2018

nh2 commented Oct 25, 2018

dtzWill commented Oct 25, 2018

dtzWill commented Oct 25, 2018

nh2 commented Oct 25, 2018

nh2 commented Oct 25, 2018

nh2 commented Oct 25, 2018

nh2 commented Oct 25, 2018

nh2 commented Oct 25, 2018

dtzWill commented Oct 25, 2018 via email

nh2 commented Oct 25, 2018

albertov commented Oct 25, 2018

dtzWill commented Oct 25, 2018 via email

albertov commented Oct 25, 2018

nh2 commented Oct 25, 2018

albertov commented Oct 26, 2018

albertov commented Oct 26, 2018 • edited Loading

albertov commented Oct 26, 2018

nh2 commented Oct 26, 2018

albertov commented Oct 28, 2018 • edited Loading

albertov commented Oct 28, 2018

matthewbauer commented Oct 31, 2018 • edited Loading

nh2 commented Nov 3, 2018

matthewbauer commented Nov 3, 2018

nh2 commented Nov 4, 2018 • edited Loading

albertov commented Oct 24, 2018 •

edited

Loading

albertov commented Oct 26, 2018 •

edited

Loading

albertov commented Oct 28, 2018 •

edited

Loading

matthewbauer commented Oct 31, 2018 •

edited

Loading

nh2 commented Nov 4, 2018 •

edited

Loading