cmd/go: Make it easier to embed compiled git SHA into binary

[kevinburke]

If I compile Go from source and run go version, I get output like this:

go version devel +475d92ba4d Thu Oct 5 10:50:18 2017 +0000 darwin/amd64

I would love to get the same output in my programs. Specifically, I'm trying to debug a reported error in a program I wrote and it would be a lot easier if I knew which SHA they were running.

It's easy in the Go build tool because it can specify what code runs during the build process. From findgoversion():

	// Otherwise, use Git.
	// What is the current branch?
	branch := chomp(run(goroot, CheckExit, "git", "rev-parse", "--abbrev-ref", "HEAD"))

	// What are the tags along the current branch?
	tag := "devel"
	precise := false

	// If we're on a release branch, use the closest matching tag
	// that is on the release branch (and not on the master branch).
	if strings.HasPrefix(branch, "release-branch.") {
		tag, precise = branchtag(branch)
	}

	if !precise {
		// Tag does not point at HEAD; add hash and date to version.
		tag += chomp(run(goroot, CheckExit, "git", "log", "-n", "1", "--format=format: +%h %cd", "HEAD"))
	}

I have options for replicating this behavior, none very good:

• Use linker flags: This blog post describes compiling Go with -ldflags -X in such a way that you can embed the SHA at compile time. https://medium.com/@joshroppo/setting-go-1-5-variables-at-compile-time-for-versioning-5b30a965d33e

  I can do this of course, but I'm not sure it's feasible or possible to ask all of my users to remember to do this when they compile my program. I can add a Make target or something to automate the instructions but that's something they have to remember to do too.

  I could release compiled libraries but that's infeasible in a company environment where tip is changing rapidly and users are used to compiling from source.

• Tag a new release every time I add a commit to the program. I wrote this tool to do that: github.com/Shyp/bump_version. Generally though it's not practical to tag every single commit, the same reason there aren't tags for every commit between Go 1.8.3 and Go 1.9.0.

It would be nice if there was some API for detecting or automatically embedding the git SHA in the binary so I can access it at runtime, and print it out when users of my program run <mybinary> version or <mybinary> env, the same way the Go team found this information useful and found a way to embed it while the program was being compiled.

[mvdan]

I have thought the same in the past, and I agree that the two current workarounds you listed are suboptimal. Let me play the devil's advocate, however:

• What happens when there is no source control, or when there is but it's not Git?
• Should this affect all packages, or just main ones?
• Would this "version hash" be taken into account to know if a package needs rebuilding?
• Could renaming a tag or removing .git break a reproducible build?

[kevinburke]

Here are places where this field would come in handy. I will update this list as I come across more examples.

• https://github.com/dominikh/go-tools/blob/master/version/version.go#L13
• https://github.com/gruntwork-io/terragrunt/blob/master/main.go#L13. Prints "terragrunt version " if you run go install ./... with no flags.

[ianlancetaylor]

@kevinburke As you say above, you can of course use -ldflags=-X today. But it seems that you aren't happy with that, because you want it to be automatic. But I really don't think we want any such thing to be automatic; as @mvdan says, what if the build doesn't use git? So it has to be under the control of some option--in which case why not just use -X?

Do you have any suggestions for how this could work in practice?

Also, one of your reasons is so that you can know which git revision a user is running. But their git SHA doesn't necessarily mean anything to you, depending on how the sources have moved around. So what you want to know is actually the git SHA that they started from. You could do that by storing the git SHA in your own repo, somehow, though I agree that that too has problems.

Does the new go.mod file used by vgo help in any way here?

[fd0]

A small note explaining what we do in practice right now for restic: we're using a small program called build.go. It allows compiling restic from a checkout of the repo and from the extracted release tar.gz without having to setup a GOPATH and only depending on the Go compiler, so it's very end-user compatible. It requires vendoring all dependencies though.

In addition (that's why I'm mentioning it here), it embeds the Git version if available and the contents of the VERSION file in the source into the binary via ldflags, so restic version can print some nice output:

restic 0.8.3 (v0.8.3-109-gd49502a5) compiled with go1.10 on linux/amd64

The first string 0.8.3 is the content of the file VERSION, followed by the output of git describe.

It turned out to be very helpful when users report issues and bugs, we request this information in our issue template.

To everybody having a similar requirements, I encourage you to give build.go a try. Just copy it to your project, configure the section at the top and it should mostly work. To all others: please ignore this comment ;)

[bcmills]

As of Go 1.12, you can get the versions of all modules compiled into the binary (in module mode) using runtime/debug.BuildInfo.

If the version in use is a pseudo-version, it also includes the first dozen or so characters of the VCS commit ID.

[fd0]

@bcmills how does this work? I've built a small test program, running it with Go 1.12beta2 shows the versions for all dependencies, but not for the main module (when compiled from a checkout of the repo):

$ go version
go version go1.12beta2 linux/amd64

$ unset GOPATH
$ export GO111MODULE=on

$ go build
[...]

$ ./godebugbuildinfo
test program for debug.BuildInfo
path: github.com/fd0/godebugbuildinfo
  main: debug.Module{Path:"github.com/fd0/godebugbuildinfo", Version:"(devel)", Sum:"", Replace:(*debug.Module)(nil)}
  deps:
    &debug.Module{Path:"github.com/fatih/color", Version:"v1.7.0", Sum:"h1:DkWD4oS2D8LGGgTQ6IvwJJXSL5Vp2ffcQg58nFV38Ys=", Replace:(*debug.Module)(nil)}
    &debug.Module{Path:"github.com/mattn/go-colorable", Version:"v0.0.9", Sum:"h1:UVL0vNpWh04HeJXV0KLcaT7r06gOH2l4OW6ddYRUIY4=", Replace:(*debug.Module)(nil)}
    &debug.Module{Path:"github.com/mattn/go-isatty", Version:"v0.0.4", Sum:"h1:bnP0vzxcAdeI1zdubAl5PjU6zsERjGZb7raWodagDYs=", Replace:(*debug.Module)(nil)}

I can only see (devel) as the version for the main module, even when I'm adding a version tag in the repository (like v1.0.0).

Is there anything I'm missing?

[dominikh]

I think this is affected by #29228. I'll reopen this issue until that's resolved.

[fd0]

Thanks!

[mvdan]

I'm pretty sure this is now being tracked under #29814, which brings up all the implementation details. I believe @bcmills was trying to consolidate all the older related issues into a single one with a proposed solution.

[dominikh]

Sounds good enough to me, now that these two issues reference each other.