runtime: go 1.6 segfault with musl libc

[nathanejohnson]

I am attempting to compile go 1.6 from a go 1.4.3 bootstrap. The build environment is Alpine linux 3.3.1 with the vanilla (i.e., no grsec) 4.1.15 kernel. The libc is musl. Go gets compiled using the build.bash script, and no errors are thrown. However, the resulting binary gives a segmentation fault whenever I attempt to use it.

abuildx86:~/aports/community/go/src/go/bin$ strace ./go
execve("./go", ["./go"], [/* 18 vars */]) = 0
set_thread_area({entry_number:-1, base_addr:0xb7768844, limit:1048575, seg_32bit:1, contents:0,      read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0 (entry_number:6)
set_tid_address(0xb7768860)             = 29565
mprotect(0xb7767000, 4096, PROT_READ)   = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0} ---
+++ killed by SIGSEGV +++
Segmentation fault

abuildx86:~/aports/community/go/src/go/bin$ uname -a
Linux abuildx86 4.1.15 #1-Alpine SMP Tue Dec 15 09:29:30 GMT 2015 i686 Linux
abuildx86:~/aports/community/go/src/go/bin$ ldd go
    /lib/ld-musl-i386.so.1 (0x80001000)
    libc.musl-x86.so.1 => /lib/ld-musl-i386.so.1 (0x80001000)
abuildx86:~/aports/community/go/src/go/bin$ 

This same environment has no problems compiling go 1.5.3 from source. My other alpine 3.3.1 VM with amd64 also has no issues compiling / running 1.5.3 or 1.6, even with the grsec kernel.

Any thoughts?

[ianlancetaylor]

Looks like a nil pointer dereference, but we don't know where it is happening. Can you run the executable under gdb, and see what it says about where the signal occurs? Try to get a backtrace.

[minux]

interesting.

could you please run go under gdb and see where does it crash?

I wonder if this is related to musl libc. Does gofmt (which should be a
static binary) crash?

The ldd output looks wrong too. At least bin/go should link to libc.so
and libpthread.so.

@bradfitz, we might need to run a musl-based builder.

[nathanejohnson]

abuildx86:~/aports/community/go/src/go/bin$ gdb ./go
GNU gdb (GDB) 7.10.1
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i586-alpine-linux-musl".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./go...done.
warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
of file /home/njohnson/aports/community/go/src/go/bin/go.
Use `info auto-load python-scripts [REGEXP]' to list them.
(gdb) run
Starting program: /home/njohnson/aports/community/go/src/go/bin/go 

Program received signal SIGSEGV, Segmentation fault.
0x00000000 in ?? ()
(gdb) 

[nathanejohnson]

gofmt is fine, but statically linked.

abuildx86:~/aports/community/go/src/go/bin$ file gofmt
gofmt: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, not stripped
abuildx86:~/aports/community/go/src/go/bin$ 

[dominikh]

The ldd output looks wrong too. At least bin/go should link to libc.so
and libpthread.so.

I can't spot anything wrong. With musl, libpthread, like other libraries, are all bundled into a single library. That includes the dynamic linker: http://wiki.musl-libc.org/wiki/Design_Concepts#Unified_libc.2Flibpthread.2Fldso

However, I've just tested Go 1.6 on the musl version of Void Linux and cannot reproduce the segfault. It is a 64 bit machine, unlike OP's 32 bit.

[nathanejohnson]

dominikh, thanks for confirming. I cannot reproduce this on amd64 either, same kernel version on same distribution with same version musl.

[minux]

Ahh, thanks for the info. Is there a musl i386 VM image that I can use to reproduce this?

[nathanejohnson]

minux, alpine 386 image would be appropriate I should think: http://www.alpinelinux.org/downloads/

get the non 64 bit one. of course,that will have grsec, and will make debugging harder . but once it installs you can install the vanilla kernel

[minux]

Thanks. Reproduced.

[minux]

I tried two programs.

bin/go fails in runtime.sched_getaffinity (the very first syscall):
(gdb) r
Starting program: /root/go.git/bin/go

Program received signal SIGSEGV, Segmentation fault.
0x00000000 in ?? ()
(gdb) bt
#0 0x00000000 in ?? ()
#1 0x0810cfa8 in runtime.sched_getaffinity () at
/root/go.git/src/runtime/sys_linux_386.s:483
#2 0x080e15df in runtime.getproccount (~r0=-1073742492)
at /root/go.git/src/runtime/os1_linux.go:88
#3 0x080e180a in runtime.osinit () at
/root/go.git/src/runtime/os1_linux.go:172
#4 0x08109e13 in runtime.rt0_go () at
/root/go.git/src/runtime/asm_386.s:147
#5 0x00000001 in ?? ()
#6 0xbffffd64 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

It's because musl doesn't put vsyscall entry point in %gs:0x10.

An externally linked hello world program couldn't even start:
(gdb) r
Starting program: /root/go.git/src/hello

Program received signal SIGSEGV, Segmentation fault.
0xb7f98656 in do_relocs (dso=dso@entry=0xb7ffd660 , rel=0x80000910,
rel_size=145784, stride=2)
at src/ldso/dynlink.c:407
407 src/ldso/dynlink.c: No such file or directory.
(gdb) bt
#0 0xb7f98656 in do_relocs (dso=dso@entry=0xb7ffd660 ,
rel=0x80000910, rel_size=145784,
stride=2) at src/ldso/dynlink.c:407
#1 0xb7f98ea7 in reloc_all (p=p@entry=0xb7ffd660 ) at
src/ldso/dynlink.c:1115
#2 0xb7f9a5e3 in __dls3 (sp=0xbffffd60) at src/ldso/dynlink.c:1598
#3 0xb7f99e5c in __dls2 (base=0xb7f7b000 "\177ELF\001\001\001",
sp=0xbffffd60)
at src/ldso/dynlink.c:1389
#4 0xb7f97ec5 in _dlstart_c (sp=0xbffffd60, dynv=0xb7ffcef8) at
src/ldso/dlstart.c:147
#5 0xb7f97dbb in _dlstart () from /lib/ld-musl-i386.so.1

Something is wrong with the relocations.

[minux]

Forked the external linking problem (also affects cgo programs) to #14478, so that we can track milestones for these two issues separately. This issue makes Go 1.6 practically unusable on linux/386 with musl libc, so it's definitely Go1.6.1 candidate. But I'm not sure #14478 is as critical.

[gopherbot]

CL https://golang.org/cl/19833 mentions this issue.

[cyli]

Sorry, I think I'm late to the game, but I may also seeing this on linux/amd64? I'm trying to build https://github.com/mattes/migrate (which depends upon https://github.com/mattn/go-sqlite3, which has some CGO, but I don't know what to look for), and everything builds fine, but I'm getting segfaults as well by doing:

docker run -it golang:1.6.0-alpine sh
apk add --update git alpine-sdk
go get github.com/mattes/migrate
migrate

The 1.6.0-alpine image is based on alpine3.3, so maybe this can be replicated using http://wiki.alpinelinux.org/cgi-bin/dl.cgi/v3.3/releases/x86_64/alpine-3.3.1-x86_64.iso?

Here are the strace and GDB results:

$ strace /go/bin/migrate 
execve("/go/bin/migrate", ["/go/bin/migrate"], [/* 14 vars */]) = 0
arch_prctl(ARCH_SET_FS, 0x7fac21441da8) = 0
set_tid_address(0x7fac21441de0)         = 12
mprotect(0x7fac21440000, 4096, PROT_READ) = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_ACCERR, si_addr=0x55961ac54f00} ---
+++ killed by SIGSEGV +++
Segmentation fault

Output from running gdb /go/bin/migrate:

Reading symbols from /go/bin/migrate...done.
Loading Go Runtime support.
(gdb) r
Starting program: /go/bin/migrate 

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7d93531 in ?? () from /lib/ld-musl-x86_64.so.1
(gdb) bt
#0  0x00007ffff7d93531 in ?? () from /lib/ld-musl-x86_64.so.1
#1  0x0000000000000000 in ?? ()
(gdb) quit

(See mattes/migrate#76 (comment))

[ianlancetaylor]

@cyli The fix for this issue is on tip but is not in 1.6.

If you have a different problem, please open a new issue. This one is closed. Thanks.

[cyli]

@ianlancetaylor The fix on tip seems to be for 386 only and I am seeing this in amd64. I am happy to open a new issue though!

[minux]

please file a new issue for amd64. thanks.

[gopherbot]

CL https://golang.org/cl/22038 mentions this issue.

[sheldonkwok]

Is this supposed to be resolved in 1.6.2? I'm still facing similar segfault issues when using go-sqlite3.

[ianlancetaylor]

@sheldonkwok This issue is believed to be fixed in 1.6.2. If you are having trouble, please open a new issue. A segfault can have many possible causes.