cmd/link: unknown symbol __stack_chk_fail_local on Alpine Linux i386

[nmeum]

What version of Go are you using (go version)?

$ go version
go version devel go1.21-02d8ebda83 Mon Feb 6 22:13:07 2023 +0000 linux/386

Does this issue reproduce with the latest release?

Yes, though for different reasons.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="386"
GOBIN=""
GOCACHE="/home/buildozer/.cache/go-build"
GOENV="/home/buildozer/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="386"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH=""
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/buildozer/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/buildozer/go/pkg/tool/linux_386"
GOVCS=""
GOVERSION="devel go1.21-02d8ebda83 Mon Feb 6 22:13:07 2023 +0000"
GCCGO="gccgo"
GO386="sse2"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m32 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build887839373=/tmp/go-build -gno-record-gcc-switches"

What did you do?

With current Git HEAD (or any commit after 066b780) on an i386 system running Alpine Linux:

1. Compiled Go using ./make.bash
2. Ran the test suite using ./run.bash

What did you expect to see?

A successful pass of the test suite.

What did you see instead?

Multiple test failures with the following error message:

# crypto/ecdsa.test
runtime/cgo(.text): unknown symbol __stack_chk_fail_local in pcrel
runtime/cgo(.text): unknown symbol __stack_chk_fail_local in pcrel
runtime/cgo(.text): unknown symbol __stack_chk_fail_local in pcrel
runtime/cgo(.text): unknown symbol __stack_chk_fail_local in pcrel
runtime/cgo(.text): unknown symbol __stack_chk_fail_local in pcrel
runtime/cgo(.text): relocation target __stack_chk_fail_local not defined
runtime/cgo(.text): relocation target __stack_chk_fail_local not defined

Why does this happen?

Commit 066b780 (CC: @thanm) removed the -fno-stack-protector C compiler flag from the cgo runtime (see #52919 and #54313 for more information on why this was needed in the first place). Instead, the commit modified cmd/link to also link against libc_nonshared.a if __stack_chk_fail_local is an unresolved symbol. This is needed on i386 as GCC emits local calls to __stack_chk_fail_local on this architecture but not on 64-bit architectures, hence this issue can only be reproduced on i386. Unfortunately, linking against libc_nonshared.a is not portable as there is no guarantee that __stack_chk_fail_local is defined by this library. On Alpine, the library providing this symbol is instead called libssp_nonshared.a but it may be named differently on other musl-based Linux distributions [1].

How can this be fixed?

I think it would be reasonable for 066b780 to be enhanced in a way that it also checks for libc_nonshared.a this would at least unbreak cgo on i386 Alpine Linux (though it is still not a portable solution), e.g.:

diff --git a/src/cmd/link/internal/ld/lib.go b/src/cmd/link/internal/ld/lib.go
index 3772b8ba90..63a0170375 100644
--- a/src/cmd/link/internal/ld/lib.go
+++ b/src/cmd/link/internal/ld/lib.go
@@ -643,11 +643,17 @@ func (ctxt *Link) loadlib() {
                                // and "__stack_chk_fail_local" is unresolved (most
                                // likely due to the use of -fstack-protector), try
                                // loading libc_nonshared.a to resolve it.
+                               //
+                               // On Alpine Linux (musl-based), the library providing
+                               // this symbol is called libssp_nonshared.a.
                                isunresolved := symbolsAreUnresolved(ctxt, []string{"__stack_chk_fail_local"})
                                if isunresolved[0] {
                                        if p := ctxt.findLibPath("libc_nonshared.a"); p != "none" {
                                                hostArchive(ctxt, p)
                                        }
+                                       if p := ctxt.findLibPath("libssp_nonshared.a"); p != "none" {
+                                               hostArchive(ctxt, p)
+                                       }
                                }
                        }
                }

With this patch, the majority of the test suite passes again. Unfortunately, tests performed without libgcc still fail:

##### Testing without libgcc.
# net.test
runtime/cgo(.text): unknown symbol __stack_chk_fail_local in pcrel
runtime/cgo(.text): unknown symbol __stack_chk_fail_local in pcrel
runtime/cgo(.text): unknown symbol __stack_chk_fail_local in pcrel
runtime/cgo(.text): unknown symbol __stack_chk_fail_local in pcrel
runtime/cgo(.text): unknown symbol __stack_chk_fail_local in pcrel
runtime/cgo(.text): relocation target __stack_chk_fail_local not defined
runtime/cgo(.text): relocation target __stack_chk_fail_local not defined
FAIL    net [build failed]
FAIL
2023/02/07 16:07:10 Failed: exit status 1
# os/user.test
runtime/cgo(.text): unknown symbol __stack_chk_fail_local in pcrel
runtime/cgo(.text): unknown symbol __stack_chk_fail_local in pcrel
runtime/cgo(.text): unknown symbol __stack_chk_fail_local in pcrel
runtime/cgo(.text): unknown symbol __stack_chk_fail_local in pcrel
runtime/cgo(.text): unknown symbol __stack_chk_fail_local in pcrel
runtime/cgo(.text): relocation target __stack_chk_fail_local not defined
runtime/cgo(.text): relocation target __stack_chk_fail_local not defined
FAIL    os/user [build failed]
FAIL
2023/02/07 16:07:10 Failed: exit status 1

This is because these tests are run with -linkmode=internal -libgcc=none and the code above is explicitly disabled if libgcc is none.

go/src/cmd/link/internal/ld/lib.go

Lines 631 to 652 in 780db9a

	if *flagLibGCC != "none" {
	hostArchive(ctxt, *flagLibGCC)
	// For glibc systems, the linker setup used by GCC
	// looks like
	//
	// GROUP ( /lib/x86_64-linux-gnu/libc.so.6
	// /usr/lib/x86_64-linux-gnu/libc_nonshared.a
	// AS_NEEDED ( /lib64/ld-linux-x86-64.so.2 ) )
	//
	// where libc_nonshared.a contains a small set of
	// symbols including "__stack_chk_fail_local" and a
	// few others. Thus if we are doing internal linking
	// and "__stack_chk_fail_local" is unresolved (most
	// likely due to the use of -fstack-protector), try
	// loading libc_nonshared.a to resolve it.
	isunresolved := symbolsAreUnresolved(ctxt, []string{"__stack_chk_fail_local"})
	if isunresolved[0] {
	if p := ctxt.findLibPath("libc_nonshared.a"); p != "none" {
	hostArchive(ctxt, p)
	}
	}
	}

As such, I would have expected these tests to also fail on i386 glibc-based Linux systems but wasn't able to test this hypothesis. If we change the code referenced above to also link against libc_nonshared.a / libssp_nonshared.a, even with -libgcc none, then all tests pass again on i386 Alpine Linux. I presently don't understand why the implementation does not resolve __stack_chk_fail_local with -libgcc none. Conceptually, I also don't understand why the internal linker is being used when cgo is involved, unconditionally using the external linker in conjunction with cgo should resolve all of those issues.

The final patch that makes on the tests pass again for me looks like this:

diff --git a/src/cmd/link/internal/ld/lib.go b/src/cmd/link/internal/ld/lib.go
index 3772b8ba90..baca7cd65f 100644
--- a/src/cmd/link/internal/ld/lib.go
+++ b/src/cmd/link/internal/ld/lib.go
@@ -630,24 +630,30 @@ func (ctxt *Link) loadlib() {
                        }
                        if *flagLibGCC != "none" {
                                hostArchive(ctxt, *flagLibGCC)
-                               // For glibc systems, the linker setup used by GCC
-                               // looks like
-                               //
-                               //  GROUP ( /lib/x86_64-linux-gnu/libc.so.6
-                               //      /usr/lib/x86_64-linux-gnu/libc_nonshared.a
-                               //      AS_NEEDED ( /lib64/ld-linux-x86-64.so.2 ) )
-                               //
-                               // where libc_nonshared.a contains a small set of
-                               // symbols including "__stack_chk_fail_local" and a
-                               // few others. Thus if we are doing internal linking
-                               // and "__stack_chk_fail_local" is unresolved (most
-                               // likely due to the use of -fstack-protector), try
-                               // loading libc_nonshared.a to resolve it.
-                               isunresolved := symbolsAreUnresolved(ctxt, []string{"__stack_chk_fail_local"})
-                               if isunresolved[0] {
-                                       if p := ctxt.findLibPath("libc_nonshared.a"); p != "none" {
-                                               hostArchive(ctxt, p)
-                                       }
+                       }
+                       // For glibc systems, the linker setup used by GCC
+                       // looks like
+                       //
+                       //  GROUP ( /lib/x86_64-linux-gnu/libc.so.6
+                       //      /usr/lib/x86_64-linux-gnu/libc_nonshared.a
+                       //      AS_NEEDED ( /lib64/ld-linux-x86-64.so.2 ) )
+                       //
+                       // where libc_nonshared.a contains a small set of
+                       // symbols including "__stack_chk_fail_local" and a
+                       // few others. Thus if we are doing internal linking
+                       // and "__stack_chk_fail_local" is unresolved (most
+                       // likely due to the use of -fstack-protector), try
+                       // loading libc_nonshared.a to resolve it.
+                       //
+                       // On Alpine Linux (musl-based), the library providing
+                       // this symbol is called libssp_nonshared.a.
+                       isunresolved := symbolsAreUnresolved(ctxt, []string{"__stack_chk_fail_local"})
+                       if isunresolved[0] {
+                               if p := ctxt.findLibPath("libc_nonshared.a"); p != "none" {
+                                       hostArchive(ctxt, p)
+                               }
+                               if p := ctxt.findLibPath("libssp_nonshared.a"); p != "none" {
+                                       hostArchive(ctxt, p)
                                }
                        }
                }

Related Issues

• os/user: unknown symbol __stack_chk_fail_local in pcrel on linux/386 #57261
• cmd/dist: make.bash failed on linux/386 due to unknow symbol #52919
• runtime/cgo(.text): unknown symbol __stack_chk_fail_local in pcrel (regression in 1.19 when building for i686) #54313

[thanm]

Thanks for the report. I'll take a look.

On Alpine, the library providing this symbol is instead called libssp_nonshared.a

If this is the case, I am wondering why we aren't seeing similar failures on our existing Alpine amd64 builder?

[nmeum]

If this is the case, I am wondering why we aren't seeing similar failures on our existing Alpine amd64 builder?

This only happens on i386 as only on i386 (and 32-bit ppc) GCC emits local calls to __stack_chk_fail_local and can thus not resolve these symbols via libc.so. This is not the case on amd64. Also, this does not only apply to musl but also to glibc. See the musl ML post referenced in my original issue description [1]:

this happens because on i386 and powerpc gcc emits local calls
to __stack_chk_fail_local which means it has to be defined within
the same module that you are linking (not in libc.so).
(that symbol should either call the extern __stack_chk_fail in
libc.so or just crash)

this should have been done by libgcc.a having a definition for
this symbol (since the compiler is using it), but instead it got
added to glibc libc_nonshared.a which is added to the link command
by using a linker script in place of libc.so.

[gopherbot]

Change https://go.dev/cl/466936 mentions this issue: cmd/link: try libspp_nonshared.a when looking for "__stack_chk_fail_local"

[gopherbot]

Change https://go.dev/cl/466935 mentions this issue: cmd/cgo: add -fno-stack-protector to CFLAGS (again)