x/vgo: vgo build should use the existing working build for projects

[cstockton]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.10.3 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

linux amd64

What did you do?

Project is internal, but I created a minimal reproduction repo for example purposes at cstockton/vgo-issue.

git clone https://github.com/cstockton/vgo-issue.git
make

What did you expect to see?

The project compile with the version of etcd I had checked out as well as only downloading the dependencies needed, i.e.:

cd src/acme/cmd/acme && go build -o acme-go # Get a baseline
cd src/acme/cmd/acme && vgo build -o acme-vgo
> vgo: downloading github.com/golang/protobuf/proto master
> vgo: downloading github.com/gogo/protobuf/gogoproto master
> vgo: downloading github.com/coreos/etcd v3.3.7 

What did you see instead?

Selects etcd from 4 years ago and begins trying to download repos that don't exist.

cd src/acme/cmd/acme && go build -o acme-go # Get a baseline
cd src/acme/cmd/acme && vgo build -o acme-vgo
vgo: resolving import "github.com/coreos/etcd/raft"
vgo: lookup code.google.com/p/gogoprotobuf/proto: unknown module code.google.com/p/gogoprotobuf/proto: no matching go-import tags
vgo: stat golang.org/x/net@c5a46024776ec35eb562fa9226968b9d543bb13a: git fetch --depth=1 https://go.googlesource.com/net c5a46024776ec35eb562fa9226968b9d543bb13a in /ws/vgo-issue/src/v/cache/vcswork/474b1070052aefe977c3c9c629be9cca5651cab1292b48693dc69b35f970d216: exit status 128:
	fatal: expected shallow/unshallow, got ERR internal server error
	fatal: The remote end hung up unexpectedly
vgo: finding github.com/coreos/etcd (latest)
vgo: adding github.com/coreos/etcd v0.5.0-alpha.5
vgo: finding github.com/coreos/etcd v0.5.0-alpha.5
vgo: lookup code.google.com/p/gogoprotobuf/proto: unknown module code.google.com/p/gogoprotobuf/proto: no matching go-import tags
vgo: stat golang.org/x/net@c5a46024776ec35eb562fa9226968b9d543bb13a: git fetch --depth=1 https://go.googlesource.com/net c5a46024776ec35eb562fa9226968b9d543bb13a in /ws/vgo-issue/src/v/cache/vcswork/474b1070052aefe977c3c9c629be9cca5651cab1292b48693dc69b35f970d216: exit status 128:
	fatal: expected shallow/unshallow, got ERR internal server error
	fatal: The remote end hung up unexpectedly
vgo: finding github.com/stretchr/testify v0.0.0-20140128171827-9cc77fa25329
vgo: finding github.com/jonboulle/clockwork v0.0.0-20141017032234-72f9bd7c4e0c
vgo: finding github.com/coreos/go-etcd v0.0.0-20141016175348-6fe04d580dfb
vgo: finding github.com/codegangsta/cli v0.0.0-20140923033435-f7ebb761e83e
vgo: downloading github.com/coreos/etcd v0.5.0-alpha.5

I wrote a similar issue in 1891 for dep - I feel there has been a heavy amount of talk and effort around algorithms for version selection and those are important. But for a vast majority of projects migrating I don't think there is a better version than what the project is currently compiling with today. Both dep and vgo are discarding all the context provided by the current workspace, I think they should use it if available and then fall back to version selection.

I don't understand all the details of the version selection algorithm I read in the blog posts so I can't comment from a technical position, I can only report as an end user that I believe selecting a 4 year old alpha version of a repo is a bit unexpected.

[myitcv]

@cstockton the issue here is similar to #25674 (comment).

At the time of writing, as you point out, the latest release of github.com/coreos/etcd is v3.3.7.

You will notice however that etcd has not yet been converted to being a Go module (no go.mod in the root of the repo, and no submodules). Hence for now we need to resort to an explicit vgo get:

cd `mktemp -d`
export GOPATH=$PWD
mkdir hello
cd hello
echo > go.mod
cat <<EOD > hello.go
package main // import "example.com/hello"

import (
        "fmt"

        "github.com/coreos/etcd/raft"
)

func main() {
        _ = raft.NewMemoryStorage()
        fmt.Println("Project compiled")
}
EOD
vgo get github.com/coreos/[email protected]
vgo build

I'll close this for now because there is nothing untoward going on here, but please do shout if I've missed anything.

[cstockton]

Thanks @myitcv, I'll give your suggestion a try in the future. I do feel like building projects has been solved for in the least common use case: a clean code base or projects that already use an existing package manager. Well we have 8 years of software out there today that was written without any of these things, I personally never moved to dep because of issues early on and I was waiting for the Go team to full buy into a solution. I feel as an end user experience, nothing could be better than giving the user a starting point of.. their current starting point. Then providing a sub command to take a path forward, this lets the user immediately migrate and start enjoying reproducible builds and slowly upgrade dependencies in a controlled way (at that point the version selection algorithms are really useful). Maybe I'm the minority though I don't know.

This means using a vgo migrate and a vgo update [--all, -a false] [pkg...] with either a vgo list or vgo update with a --dry-run flag. This covers the way I want to migrate to a new way of writing Go code, which is the way I want to migrate to a new way of literally anything. Small, measurable steps. Right now switching to vgo or dep means changing every single dependency which introduces failed compilations for a process that is unfamiliar to you. Far worst would be the cases when the upgrades allow compilation but introduce a production only bug, anyone first suspicion would be "vgo broke my code!" "it was working fine before vgo!".

Here is what would have been an amazing experience in my opinion:

cd $GOPATH/acme
vgo migrate
...
> wrote go.mod with:
> $GOPATH/src/... @vN.N.N # current tag I am using to build my project today.
> $GOPATH/src/... @e0e0e0 # current commit hash I am using to build my project today.

Now the first thing I am going to do is check for a reproducible build from a fresh checkout:

mkdir -p tmp/{,src} && cd tmp
export GOPATH=$(pwd)
cd src && git clone acme && cd acme && vgo build
# run my tests

Fantastic, now I have reproducible builds and I can work package upgrades into future sprints so I can spend dedicated time for big dependencies like etcd/raft are very critical to my system:

cd $GOPATH/src/acme
vgo update --dry-run github.com/coreos/etcd
> here's what will change if you go through with this:
>   etcd @ <tag>
>   dep1
>   dep2

Eventually my initial migration state will have caught up to what vgo is designed for and all is well. The experience would be seamless and pain free. I understand the caveats of checking the current work space, I get there is only so far you can go in some cases. Maybe repos that don't have git information in the vendor directory for example, could you still determine the version? Yes. Maybe not worth the compute cycles to do so though (iterate up / down comparing files and then hashes from closest revision matching the same set of files or something? not sure), but maybe it is.

Maybe this is unreasonable and I am a small minority, I know there is nothing stopping me from writing a tool that does this process and I just might do that. To be clear this isn't a proposal, consider it a detailed experience report from a decent sized internal project with >50 dependencies to maybe stir some thought. Thanks for the efforts spent on vgo thus far, I feel the entire Go ecosystem will benefit greatly once it is in wide spread use. I just hope the migration process becomes less painful, because the quicker everyone migrates the better the experience for everyone will be.