plugin: package can't open plugin built with same Go runtime

[frankgreco]

go version go1.8.3 linux/amd64

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build655969460=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

Received the following error when trying to open a plugin:

plugin.Open: plugin was built with a different version of package errors

Now, this can happen when:

We should explicitly reject an attempt to open a plugin that was created by a different version of the Go toolchain.

However, here are reproducible steps that show that this is happening with the same version of go

$ git clone [email protected]:northwesternmutual/kanali.git && cd kanali
$ git checkout plugin-open
$ minikube start
$ ./scripts/install.sh # wait until all pods are in running state
$ kubectl apply -f ./examples/exampleSeven.yaml
$ curl --header 'apikey: 0HfVWylwxchODd3s4A7D9Zoel0Xo83iQ' --insecure $(minikube service kanali --url --format="https://{{.IP}}:{{.Port}}")/api/v1/example-seven

Here is the Dockerfile being built above. As you can see, the same go installation is used to both compile both the program and the plugin:

ARG GO_VERSION=1.8.3
FROM golang:${GO_VERSION} AS build-stage
MAINTAINER [email protected]
ARG VERSION="unknown version"
ARG GLIDE_VERSION=0.12.3
WORKDIR /go/src/github.com/northwesternmutual/kanali/
COPY glide.lock glide.yaml Makefile /go/src/github.com/northwesternmutual/kanali/
RUN wget "https://github.com/Masterminds/glide/releases/download/v${GLIDE_VERSION}/glide-v${GLIDE_VERSION}-`go env GOHOSTOS`-`go env GOHOSTARCH`.tar.gz" -O /tmp/glide.tar.gz \
    && mkdir /tmp/glide \
    && tar --directory=/tmp/glide -xvf /tmp/glide.tar.gz \
    && rm -rf /tmp/glide.tar.gz
RUN export PATH=$PATH:/tmp/glide/`go env GOHOSTOS`-`go env GOHOSTARCH` \
    && make install
COPY ./ /go/src/github.com/northwesternmutual/kanali/
RUN sed -ie "s/changeme/`echo ${VERSION}`/g" /go/src/github.com/northwesternmutual/kanali/cmd/version.go
RUN curl -O https://raw.githubusercontent.com/northwesternmutual/kanali-plugin-apikey/master/plugin.go
RUN go build -buildmode=plugin -o apiKey.so plugin.go
RUN make build

FROM centos:latest
MAINTAINER [email protected]
RUN curl http://curl.haxx.se/ca/cacert.pem -o /etc/pki/tls/certs/ca-bundle.crt
COPY --from=build-stage /go/src/github.com/northwesternmutual/kanali/apiKey.so ./apiKey.so
COPY --from=build-stage /go/src/github.com/northwesternmutual/kanali/kanali .
ENTRYPOINT ["/kanali"]

[ianlancetaylor]

CC @crawshaw

[crawshaw]

There are too many layers here for me to dig through your example, sorry.

Is it possible that you have two different copies of package errors? Maybe the plugin is built with an older or newer copy?

[robmccoll]

So here is a simpler reproduction of the problem from Go 1.9 (we were not experiencing problems in 1.8.3):

• Download and extract Go 1.9 (using Linux x86_64 here)
• Create main.go and plugin.go following the example from the documentation for the plugin package, but add a needless import of the plugin package to the plugin:

// main.go
package main

import "plugin"

func main() {
  p, err := plugin.Open("plugin.so")
  if err != nil {
    panic(err)
  }
  v, err := p.Lookup("V")
  if err != nil {
    panic(err)
  }
  f, err := p.Lookup("F")
  if err != nil {
    panic(err)
  }
  *v.(*int) = 7
  f.(func())() // prints "Hello, number 7"
}

// plugin.go
package main

import "fmt"

// XXX this is weird
import _ "plugin"

var V int

func F() { fmt.Printf("Hello, number %d\n", V) }

• Build these and run main:

$go build -buildmode=plugin plugin.go 
$go build main.go
$./main
panic: plugin.Open: plugin was built with a different version of package plugin

goroutine 1 [running]:
main.main()
	/home/rmccoll/tmp/main.go:8 +0x3bb

• Now go into the go src for runtime and plugin and force them to rebuild:

cd /usr/local/go/src/runtime && go install -a
cd /usr/local/go/src/plugin && go install -a

• Rebuild your plugin and main and run again:

$go build -buildmode=plugin plugin.go 
$go build main.go
$./main
Hello, number 7

• Scratch head for a while...

[crawshaw]

Thanks for the reproduction, that gives me an idea what's happening.

Could you try patching your Go installation with https://golang.org/cl/61071 and rebuilding with make.bash?

That CL doesn't look relevant, but I believe because one copy of the plugin package is built with -gcflags=-shared and one is built without, they have different version hashes, which is why you're seeing that error.

(Unrelated to this issue: if the bugs I've been hunting down with CL 61071 turn out "just" to be version mismatches, maybe there's some other solution to that problem.)

[crawshaw]

My guess about what's happening: the C/C++ compiler used by cgo to build the version of Go 1.9 you downloaded is different from your local C/C++ compiler.

When you build the main binary, it uses the shipped .a files.

When you build the plugin, it gets built with -shared, which requires building different .a files than the ones that came in the release, so your C/C++ compiler is invoked. The C/C++ object code in the plugin package is slightly different, so a different version hash is computed.

The only general solution to this is relaxing the versioning scheme we are using. (For example, if we hashed exported APIs this wouldn't be a problem.)

[crawshaw]

Hmm, that explanation isn't wholly satisfying, because the part of the PKGDEF being used to generate the version doesn't seem to obviously include any mention of the cgo parts of the package.

[crawshaw]

@robmccoll I have simply been overanalyzing this. There is something off about the .a files that come with the Go 1.9 binary distribution.

If you download it, then use ar x to extract the __.PKGDEF from $GOROOT/pkg/linux_amd64/plugin.a (the contents of __.PKGDEF is used to build the package version for plugins) you get one piece of binary export data. Then run go install -a std and do it again, and you get another piece of binary export data.

The variants -installsuffix=shared -gcflags=-shared and -installsuffix=dynlink -gcflags=-dynlink produce the same binary export data as the default install. The odd file out comes with the binary distribution.

I'll work out how to unpack it and see if I can spot the difference. For now, the quick fix is go install -a std.

[crawshaw]

@griesemer, @alandonovan do you have any tools for pretty printing the binary export format? I've started hacking up something using go/types, but I think it may be hiding the the minor differences between these files.

[ianlancetaylor]

@mdempsky see previous comment.

[crawshaw]

Staring at the bytes revealed that the binary export data has an absolute path in it from where the 1.9 tarball was built:

/tmp/workdir/go..._plugin_dlopen.go

when I build the .a files again with go install -a std, it gets replaced with the absolute path to my workstation's GOROOT.

Presumably this has something to do with cgo. Still reading code.

[crawshaw]

So it looks like absolute paths in the binary export position information is deliberate (there are calls to AbsFilename in cmd/compile's bexport.go).

Is this intended, @griesemer, @mdempsky? If so, do we not care about reproducible builds for intermediate .a files?

Regardless, we need a new strategy for -buildmode=shared and -buildmode=plugin, which use a hash of the export data for versioning. With absolute paths, builds cannot reproduce, and plugin versions are no use.

One option would be for the compiler could generate the hash for the linker, from the export data minus the position information.

Another option would be for the linker could learn to parse the export data and do it. This is more work, but may be a useful first step towards generating more useful versioning information for the non-default build modes.

[robmccoll]

Huh - sorry I missed the updates so far. For now, we've just added rebuilding the standard library as part of the scripting for setting up our build environment.

I'm curious - how were the package hash contents chosen? Were the paths really intended to be paths or is that just the overlap between the filesystem and import paths? What would be the impact of not including it?

[gopherbot]

Change https://golang.org/cl/63693 mentions this issue: cmd/compile: replace GOROOT in //line directives

[mastersingh24]

Is this slated for 1.9.2?

[crawshaw]

CL 63693 is slated for 1.9.2 because of #21825.

[gopherbot]

Change https://golang.org/cl/70975 mentions this issue: [release-branch.go1.9] cmd/compile: replace GOROOT in //line directives