make rebuild no longer works

[djmitche]

dustin@lamport ~/go/src/github.com/taskcluster/taskcluster-worker [master] $ make rebuild
go fmt $(go list ./... | grep -v /vendor/)
CGO_ENABLED=0 go install
go install os/user: open /usr/local/go/pkg/linux_amd64/os/user.a: permission denied
go install net: open /usr/local/go/pkg/linux_amd64/net.a: permission denied
Makefile:28: recipe for target 'build' failed
make: *** [build] Error 1

[petemoore]

@djmitche what output do you get from go env?

[petemoore]

Note, since adding more comprehensive linting in #179, it was necessary to switch from go build to go install in the make file, since some of the linters require the compiled object files to operate.

For example, I can imagine if you don't have write access to your $GOPATH/bin or $GOPATH/pkg directory, you may hit a problem. Perhaps your GOPATH is set to your system go installation (i.e. your GOROOT)?

[djmitche]

ustin@lamport ~/go/src/github.com/taskcluster/taskcluster-worker [bug1322256] $ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/dustin/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build813643544=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

[petemoore]

@djmitche Is it possible you might have edited source files under /usr/local/go/src/os/user and /usr/local/go/src/net e.g. when stepping through code that calls into the standard library?

Can you see if there are files in those directories that are touched more recently than the others?

Maybe if you'd inadvertently modified those library files, it might try to reinstall the packages in the system go installation, and then maybe gets stuck.

Are you able to go install any code that imports net or os/user packages?

[petemoore]

Explanation: a possible cause is that the go sources in net and os/user packages in your go installation are out-of-sync with the binaries, so the go install is trying to rebuild those system packages and install them in the system installation (i.e. under /usr/local/go).

Or maybe the package files (e.g. /usr/local/go/pkg/linux_amd64/os/user.a) are just not readable by your go user? Not clear if it is a problem reading these files (which should be allowed) or writing them (which shouldn't be allowed, and could be caused by modified sources).

[djmitche]

They are readable:

-rw-r--r-- 1 root root 96398 Oct 19 17:08 /usr/local/go/pkg/linux_amd64/os/user.a

and all of the files were modified on October 19, which is probably when this Go version was installed.

[petemoore]

@djmitche Maybe a case of Dr-Terrible/go-overlay#15 ?

[petemoore]

Also, does it disappear if you remove CGO_ENABLED=0 from the line of the Makefile? I wonder if your go installation was built with cgo enabled, and this is therefore trying to rebuild your system packages with it disabled?

Other than that, I'm out of ideas. 😄

[djmitche]

That might be. I'm installing go from Tarballs, but maybe something's not right about cleaning out the old versions and adding the new. I'll try reinstalling. CGO_ENABLED=0 worked fine with `go build` before.

[djmitche]

No luck -- I reinstalled go (after rm -rf /usr/local/go), and:

dustin@lamport ~/go/src/github.com/taskcluster/taskcluster-worker [bug1322256] $ make
go fmt $(go list ./... | grep -v /vendor/)
CGO_ENABLED=0 go install
go install net: open /usr/local/go/pkg/linux_amd64/net.a: permission denied
go install os/user: open /usr/local/go/pkg/linux_amd64/os/user.a: permission denied
Makefile:28: recipe for target 'build' failed
make: *** [build] Error 1
dustin@lamport ~/go/src/github.com/taskcluster/taskcluster-worker [bug1322256] $ go version
go version go1.7.3 linux/amd64

[petemoore]

CGO_ENABLED=0 worked fine with go build before.

I believe go build won't install anything, just place a binary in the current directory, whereas go install will install any build artifacts it creates along the way. Typically this means intermediate libraries go in $GOPATH/pkg and the final executable goes in $GOPATH/bin but in the case of system library changes, I'm guessing these will get rebuilt and placed in <GOROOT>/pkg in place of the original versions. Since we switched from go build to go install I suspect this is what highlighted the problem on your environment, but perhaps if you had run go install on anything that touched those system libraries (and maybe just with CGO_ENABLED=0), I'm guessing you would have always had this problem.

I think @jonasfj might have added the CGO_ENABLED=0 - he might know why we need it. From my experience, I've disabled it when using the race detector, but that happens during test phase, not build phase, so I'm not why we want to disable it, other than it maybe produces cleaner/safer binaries?

[petemoore]

Ah - this looks interesting golang/go#18981