cmd/go: does not distinguish pseudo-versions from pre-release versions when both are included in module proxy @v/list endpoint

[dmitshur]

What version of Go are you using (go version)?

$ go version
go version go1.12.6 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

It also happens on tip, I tried with go version devel +71a6a444 (1~ day old).

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/dmitshur/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/dmitshur/go"
GOPROXY="https://proxy.golang.org"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/3m/rg2zm24d1jg40wb48wr0hdjw00jwcj/T/go-build318521526=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Consider a repository with 3 commits and a pre-release tag named v0.5.0-alpha pointing to the middle commit. I've made such a test repository at https://github.com/dmitshur-test/modtest5. So, the module with the module path github.com/dmitshur-test/modtest5 has (at least) these three versions:

1. v0.0.0-20190619020302-197a620e0c9a - pseudo-version for the 1st commit
2. v0.5.0-alpha - pre-release version for the 2nd commit
3. v0.5.0-alpha.0.20190619023908-3da23a9deb9e - pseduo-version for the 3rd commit

(Source.)

$ go list -m github.com/dmitshur-test/modtest5@197a620e0c9a7a44f03a73314220f8faccaf76bb
github.com/dmitshur-test/modtest5 v0.0.0-20190619020302-197a620e0c9a
$ go list -m github.com/dmitshur-test/modtest5@ac1e91846d66c96f83d3013692715ead23d978db
github.com/dmitshur-test/modtest5 v0.5.0-alpha
$ go list -m github.com/dmitshur-test/modtest5@3da23a9deb9e9f230e64fbae87befd59ffb46457
github.com/dmitshur-test/modtest5 v0.5.0-alpha.0.20190619023908-3da23a9deb9e

Based on the rules described in the 5th paragraph of https://golang.org/cmd/go/#hdr-Module_aware_go_get, since a pre-release tag exists, pseudo-versions are not considered when determining the module version that the module query latest resolves to. So, latest should be v0.5.0-alpha (pointing to 2nd commit) rather than a pseudo-version pointing to the 3rd commit.

This works when fetching directly:

$ go list -m -json github.com/dmitshur-test/modtest5@latest
{
	"Path": "github.com/dmitshur-test/modtest5",
	"Version": "v0.5.0-alpha",
	"Time": "2019-06-19T02:04:46Z"
}

The last paragraph in the module proxy protocol section of cmd/go documentation says:

Even when downloading directly from version control systems, the go command synthesizes explicit info, mod, and zip files and stores them in its local cache, $GOPATH/pkg/mod/cache/download, the same as if it had downloaded them directly from a proxy. The cache layout is the same as the proxy URL space, so serving $GOPATH/pkg/mod/cache/download at (or copying it to) https://example.com/proxy would let other users access those cached module versions with GOPROXY=https://example.com/proxy.

Now, if one downloads the 3 versions to a local module cache, then uses its resultant contents on disk as a module proxy implementation, the module query latest ends up incorrectly resolving to the pseduo-version v0.5.0-alpha.0.20190619023908-3da23a9deb9e instead of the pre-release version v0.5.0-alpha:

$ export GO111MODULE=on
$ export GOPROXY=direct
$ export GOPATH=$(mktemp -d)

$ go mod download github.com/dmitshur-test/[email protected]
go: finding github.com/dmitshur-test/modtest5 v0.0.0-20190619020302-197a620e0c9a
$ go mod download github.com/dmitshur-test/[email protected]
go: finding github.com/dmitshur-test/modtest5 v0.5.0-alpha
$ go mod download github.com/dmitshur-test/[email protected]
go: finding github.com/dmitshur-test/modtest5 v0.5.0-alpha.0.20190619023908-3da23a9deb9e

$ cat $GOPATH/pkg/mod/cache/download/github.com/dmitshur-test/modtest5/@v/list
v0.0.0-20190619020302-197a620e0c9a
v0.5.0-alpha
v0.5.0-alpha.0.20190619023908-3da23a9deb9e

$ export GOPROXY="file://${GOPATH}/pkg/mod/cache/download"
$ export GOPATH=$(mktemp -d)

$ go list -m -json github.com/dmitshur-test/modtest5@latest
{
	"Path": "github.com/dmitshur-test/modtest5",
	"Version": "v0.5.0-alpha.0.20190619023908-3da23a9deb9e",
	"Time": "2019-06-19T02:39:08Z"
}

What did you expect to see?

I expected pre-release version to be consistently used regardless if fetching from direct or a module proxy:

$ GOPROXY=direct go list -m github.com/dmitshur-test/modtest5@latest
github.com/dmitshur-test/modtest5 v0.5.0-alpha
$ GOPROXY=<some proxy whose list includes pseudo-versions> go list -m github.com/dmitshur-test/modtest5@latest
github.com/dmitshur-test/modtest5 v0.5.0-alpha

What did you see instead?

A pseudo-version was treated as a pre-release version and used as latest when fetching via the module proxy protocol that includes pseudo-versions in the @v/list endpoint:

$ GOPROXY=direct go list -m github.com/dmitshur-test/modtest5@latest
github.com/dmitshur-test/modtest5 v0.5.0-alpha
$ GOPROXY=<some proxy whose list includes pseudo-versions> go list -m github.com/dmitshur-test/modtest5@latest
github.com/dmitshur-test/modtest5 v0.5.0-alpha.0.20190619023908-3da23a9deb9e

---

Based on my understanding, this is a minor problem that is unlikely to affect a lot of users. I'm reporting it anyway. This issue is similar/related to #32695.

I believe a solution could be... Either it should be clarified in the module proxy protocol specification that the @v/list endpoint must not include pseudo-versions (otherwise they get treated as if they were pre-release versions), or the go command should correctly distinguish pseudo-versions from pre-release versions when both are included in the @v/list output.

/cc @jayconrod @hyangah @bcmills

[gopherbot]

Change https://golang.org/cl/183402 mentions this issue: cmd/go/internal/modfetch: filter pseudo-versions from proxy /list endpoints

[gopherbot]

Change https://golang.org/cl/183401 mentions this issue: cmd/go/testdata: update addmod.go to work with the mainline go command

[bcmills]

Given that file trees including pseudo-versions already exist in the wild, I think it's simplest to filter them out within (*modfetch.ProxyRepo).Versions. That turns out to be fairly trivial.

[gopherbot]

Change https://golang.org/cl/183619 mentions this issue: cmd/go/internal/modfetch: return structured errors from proxy operations

[gopherbot]

Change https://golang.org/cl/183845 mentions this issue: cmd/go/internal/modfetch: halt proxy fallback if the proxy returns a non-404/410 response for @latest