cmd/cgo: does not replace references to paths in GOROOT/ with $GOROOT

[orivej]

Normally Go compiles its own sources such that debug info pathnames look like $GOROOT/src/package/path.go rather than /absolute/path/to/go/src/package/path.go. However, this does not happen when compiling sources with cgo. In particular src/plugin/plugin_dlopen.go produces this temporary file plugin_dlopen.cgo1.go:

…
//line /tmp/nds-install-go_1_9/share/go/src/plugin/plugin_dlopen.go:43
func lastIndexByte(s string, c byte) int {
…

and after unpacking the resulting plugin.a (ar x plugin.a) and disassemly (go tool objdump _go_.o) it can be seen that pure Go sources are refered through $GOROOT, unlike cgo sources:

…
TEXT %22%22.Open(SB) gofile..$GOROOT/src/plugin/plugin.go
…
TEXT %22%22.lastIndexByte(SB) gofile../tmp/nds-install-go_1_9/share/go/src/plugin/plugin_dlopen.go
…

I would like the second line to be:

TEXT %22%22.lastIndexByte(SB) gofile..$GOROOT/src/plugin/plugin_dlopen.go

Here is a possible fix that replaces absolute path names in *.cgo1.go:

diff --git a/src/go/printer/printer.go b/src/go/printer/printer.go
index dbb4bbd90c..16a0b84fa5 100644
--- a/src/go/printer/printer.go
+++ b/src/go/printer/printer.go
@@ -11,6 +11,7 @@ import (
 	"go/token"
 	"io"
 	"os"
+	"runtime"
 	"strconv"
 	"strings"
 	"text/tabwriter"
@@ -207,7 +208,8 @@ func (p *printer) lineFor(pos token.Pos) int {
 func (p *printer) writeLineDirective(pos token.Position) {
 	if pos.IsValid() && (p.out.Line != pos.Line || p.out.Filename != pos.Filename) {
 		p.output = append(p.output, tabwriter.Escape) // protect '\n' in //line from tabwriter interpretation
-		p.output = append(p.output, fmt.Sprintf("//line %s:%d\n", pos.Filename, pos.Line)...)
+		filename := strings.Replace(pos.Filename, runtime.GOROOT(), "$GOROOT", -1)
+		p.output = append(p.output, fmt.Sprintf("//line %s:%d\n", filename, pos.Line)...)
 		p.output = append(p.output, tabwriter.Escape)
 		// p.out must match the //line directive
 		p.out.Filename = pos.Filename

What version of Go are you using (go version)?

go1.9

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

NixOS Linux amd64

What did you do?

I tried to build journalbeat that uses the plugin package with Go 1.9. It failed Nix check that executables should not refer to Go root via an absolute pathname.

[odeke-em]

Thank you for the report @orivej and for the proposed fix.

So we can't yet review patches on Github, but we do on Gerrit. If you are interested in submitting one, please take a look at https://golang.org/doc/contribute.html just in case you haven't yet.

Otherwise I'll also tag the cgo crew @ianlancetaylor @hirochachacha who might be interested in this issue.

[ianlancetaylor]

Changing go/printer is not going to be the right approach. go/printer is a general purpose package and it shouldn't be in the business of modifying file names. The place to change this would be cmd/cgo or cmd/go.

Also I think @crawshaw has been looking at something in this area.

[orivej]

Somehow Go 1.8 was not affected: even though its cgo files were the same, object files referred to $GOROOT/src/plugin/plugin_dlopen.go. This bug was introduced in 4808fc4.

CC @griesemer

[ianlancetaylor]

Since it worked in 1.8, tentatively setting milestone to 1.9.1. We'll see how invasive the fix is.

[crawshaw]

I experimented with a fix for this as part of #21373. It's easy enough not to use absolute paths, but I assume @griesemer introduced them for a reason, so I'll leave this bug to him.

[odeke-em]

@crawshaw, just in case 1.9.1 has a tight shipping deadline, @griesemer is out of office for the perhaps the next 6 days, at least as per his message on this review https://go-review.googlesource.com/c/go/+/59450#message-84f433cbd5a65c13bc58e251331687a87d01748f.

[orivej]

FWIW I believe that @griesemer considered $GOROOT/ paths absolute too: see this function. He must have missed application of this logic to pathnames read from //line comments.

[crawshaw]

Thanks @orivej that makes the problem in cmd/compile clear.

The Go binary distribution is built in /tmp/workdir/go. GOROOT_FINAL is set to /usr/local/go. This is the string that cmd/dist writes into cmd/internal/objabi/zbootstrap.go. This string, /usr/local/go is what AbsFile uses to replace the GOROOT with the string '$GOROOT'. But because the .a files we ship are built in /tmp/workdir/go, the string replace fails.

I believe the most robust fix is to have cmd/go pass its computed GOROOT to cmd/compile as an environment variable (if it's not already set). That will mean cmd/compile does the right thing when the initial .a files are set, and also do the right thing when the entire GOROOT is moved later to an arbitrary directory.

That will resolve the release blocker part of this issue, though you may want to keep it around to look at changing the cmd/cgo program to not write absolute paths.

[orivej]

That will resolve the release blocker part of this issue, though you may want to keep it around to look at changing the cmd/cgo program to not write absolute paths.

My initial patch was only for demonstration. The substitution of $GOROOT is logical either before or after the generation of *cgo*.go files with //line comments, and I don't mind if Go continues to prefer after.

[orivej]

This does not explain why substitution worked before 4808fc4 : it does not seem to have used the environment to pass the value of GOROOT.

[crawshaw]

Turns out cmd/go is already passing GOROOT into cmd/compile, so the value of GOROOT used in AbsFile is correct. So somehow the path coming from a //line directive is bypassing that function's logic.

[crawshaw]

Oh I see, I was misreading some code.

So bexport.go uses AbsFilename, which returns whatever string is inside PosBase. That comes from the //line directive, which is an unmangled absolute path, which is why we don't see GOROOT.

For positions not coming from a //line, those are built in tokenizer.go and use objabi.AbsFile to replace the GOROOT with '$GOROOT'.

[gopherbot]

Change https://golang.org/cl/63693 mentions this issue: cmd/compile: replace GOROOT in //line directives

[orivej]

Thank you for fixing this! I hope it will be shipped with 1.9.1.

[crawshaw]

Reopening for 1.9.1 cherry-pick.

[rsc]

CL 63693 OK for Go 1.9.2.

[gopherbot]

Change https://golang.org/cl/70975 mentions this issue: [release-branch.go1.9] cmd/compile: replace GOROOT in //line directives

[rsc]

go1.9.2 has been packaged and includes:

• CL 63693

The release is posted at golang.org/dl.

— golang.org/x/build/cmd/releasebot, Oct 26 21:09:13 UTC