cmd/go: 'mod tidy' downgrades an unused indirect dependency upgraded by 'get -u'

[SamWhited]

I thought I remembered an issue about this being fixed already, but I couldn't find it and it's still happening and causing problems. Sorry of this is noise and the issue is still floating around somewhere.

The specific dependency in this example that's being stripped out and re-added does not use modules, but does use semantic versioned tags which testify is pinning to.

What version of Go are you using (go version)?

$ go version
go version devel +a2bb68de4d1c 20190111160859 linux/amd64

Does this issue reproduce with the latest release?

It happens with go1.12beta2, go1.12beta1, and go1.11.4.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN="/home/sam/bin"
GOCACHE="/home/sam/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/sam/go"
GOPROXY=""
GORACE=""
GOROOT="/home/sam/Projects/go"
GOTMPDIR=""
GOTOOLDIR="/home/sam/Projects/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/sam/test/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build393255074=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Create a program that imports a package that exhibits this behavior. I noticed it with the Stripe SDK, and testify seems to be the dependency causing the problem so the following also has the same issue:

package main

import (
	"fmt"

	"github.com/stretchr/testify/assert"
)

func main() {
	_ = assert.CallerInfo
	fmt.Println("vim-go")
}

Now run go mod init testmod, and go get -u. You will end up with the following go.mod file:

module testmod

go 1.12

require (
	github.com/davecgh/go-spew v1.1.1 // indirect
	github.com/stretchr/objx v0.1.1 // indirect
	github.com/stretchr/testify v1.3.0
)

now run go mod tidy and you wind up with the following:

module testmod

go 1.12

require (
	github.com/davecgh/go-spew v1.1.1 // indirect
	github.com/stretchr/testify v1.3.0
)

What did you expect to see?

The file does not change.

What did you see instead?

Every time you run go mod tidy it strips out indirect dependencies inserted by go get -u and you end up with unmerged changes in your repo.

[mvdan]

/cc @bcmills

[bcmills]

Looks like they agree on the build list but disagree about whether dependencies are indirect:

$ go get -u
go: finding github.com/davecgh/go-spew v1.1.1
go: finding github.com/stretchr/objx v0.1.1
go: downloading github.com/davecgh/go-spew v1.1.1

$ cat go.mod
module golang.org/issue/scratch

require (
        github.com/davecgh/go-spew v1.1.1 // indirect
        github.com/stretchr/objx v0.1.1 // indirect
        github.com/stretchr/testify v1.3.0
)

$ go list -m all
golang.org/issue/scratch
github.com/davecgh/go-spew v1.1.1
github.com/pmezard/go-difflib v1.0.0
github.com/stretchr/objx v0.1.1
github.com/stretchr/testify v1.3.0

$ go mod tidy

$ cat go.mod
module golang.org/issue/scratch

require (
        github.com/davecgh/go-spew v1.1.1 // indirect
        github.com/stretchr/testify v1.3.0
)

$ go list -m all
golang.org/issue/scratch
github.com/davecgh/go-spew v1.1.1
github.com/pmezard/go-difflib v1.0.0
github.com/stretchr/objx v0.1.0
github.com/stretchr/testify v1.3.0

[bcmills]

Wait, no. go mod tidy ends up downgrading; that might be a bug.

[bcmills]

This case is interesting, though: nothing in the build actually imports from objx, and nothing in objx affects the versions of other libraries, so perhaps go mod tidy is correct to remove it from the go.mod file even though it remains in the transitive module graph.

[SamWhited]

This case is interesting, though: nothing in the build actually imports from objx.

The tests do though (via testify/mock), and it appears that mod tidy is adding things behind build tags and test files generally; it's just this one that it's stripping out for some reason.
Even though it doesn't affect any other libraries, if we require pinning to a specific version of objx stripping it out might mean that we pull in a newer version that's broken, no?

[bcmills]

The tests do though (via testify/mock),

Nothing in the import graph imports testify/mock. go mod tidy only considers the packages transitively imported by the main module and the transitive dependencies of the tests of those packages.

if we require pinning to a specific version of objx stripping it out might mean that we pull in a newer version that's broken, no?

In this case, go test all literally doesn't build anything in objx, so the only way it could be broken is through some invalid module requirement.

[SamWhited]

Nothing in the import graph imports testify/mock. go mod tidy only considers the packages transitively imported by the main module and the transitive dependencies of the tests of those packages.

doc.go imports testify/http which imports testify/mock in a test_round_tripper.go, unless I'm still misunderstanding. It's a weird _ import though. I don't really understand why, not having ever used testify except transitively. It doesn't appear to have any import side effects (thank goodness).

In this case, go test all literally doesn't build anything in objx, so the only way it could be broken is through some invalid module requirement.

I see; I suppose if it's an _ import and nothing is actually used that's fair.

[bcmills]

doc.go imports testify/http which imports testify/mock

Nothing in the example program imports github.com/stretchr/testify (that is, the package containing doc.go) itself.

[SamWhited]

oops, sorry, you're right of course.

[rsc]

Why did go get -u decide to look at objx?

[thanasik]

Having this issue as well, but with local private dependencies. Both go mod tidy and go mod vendor alter my go mod file this way, and running go build on the modified go.mod file results in an error during build saying cannot find module for path (though it does actually vendor it, the files are in the vendor dir). I have to manually add the dependency back in order to get the build working. For context, my go version is go version go1.12.1 linux/amd64

[bcmills]

This is probably a side-effect of #26902.

[thepudds]

This is probably a side-effect of #26902.

Now that #26902 is resolved, I tried to reproduce this using tip (e.g., go get golang.org/dl/gotip && gotip download).

As hoped or expected, github.com/stretchr/objx now no longer shows up after the initial go get -u, and then running go mod tidy has no impact:

gotip mod init testmod
cat <<EOF > main.go
package main
import (
    "fmt"
    "github.com/stretchr/testify/assert"
)
func main() {
    _ = assert.CallerInfo
    fmt.Println("vim-go")
}
EOF
gotip get -u
cp go.mod go.mod_orig
gotip mod tidy
diff go.mod_orig go.mod
cat go.mod

which results in no diff reported, and go.mod contains:

module testmod

go 1.13

require (
        github.com/davecgh/go-spew v1.1.1 // indirect
        github.com/stretchr/testify v1.3.0
)

I used go version devel +ba9bc8e5 Sun May 5 12:55:21 2019 +0000.

Also, gotip get -u all seems to give same result as gotip get -u (which seems expected as well).

I did not however attempt to evaluate all the commentary in this issue -- I only tried the first set of steps reported here (in comment #29702 (comment)).

@SamWhited would you be willing to try this with tip? Does this seem resolved from your perspective, or does it seem there are still pieces pending?

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

[colega]

This still happens on go version go1.12.6 darwin/amd64 and go version devel +d410642 Mon Jul 1 21:30:23 2019 +0000 darwin/amd64 (gotip).

We're reproducing it in the following scenario, consider two libraries github.com/zero-nine-three and github.com/one-zero-zero that import nothing but github.com/prometheus/client_golang on versions v0.9.3 and v1.0.0 respectively:

$ tail -n +1 ../zero-nine-three/go.mod ../one-zero-zero/go.mod
==> ../zero-nine-three/go.mod <==
module github.com/zero-nine-three

go 1.12

require github.com/prometheus/client_golang v0.9.3

==> ../one-zero-zero/go.mod <==
module github.com/one-zero-zero

go 1.12

require github.com/prometheus/client_golang v1.0.0

Then go mod tidy removes and go get adds the github.com/go-logfmt/logfmt v0.4.0 // indirect dependency:

$ go mod tidy

$ cat go.mod
module bug

go 1.12

replace github.com/one-zero-zero => ../one-zero-zero

replace github.com/zero-nine-three => ../zero-nine-three

require (
	github.com/one-zero-zero v0.0.0-00010101000000-000000000000
	github.com/zero-nine-three v0.0.0-00010101000000-000000000000
)

$ go get

$ cat go.mod
module bug

go 1.12

replace github.com/one-zero-zero => ../one-zero-zero

replace github.com/zero-nine-three => ../zero-nine-three

require (
	github.com/go-logfmt/logfmt v0.4.0 // indirect
	github.com/one-zero-zero v0.0.0-00010101000000-000000000000
	github.com/zero-nine-three v0.0.0-00010101000000-000000000000
)

We have noted that this behavior is not reproduced if we're importing a library without a hostname (i.e., zero-nine-three instead of github.com/zero-nine-three)

This behavior forces us to disable the tidyness check in our CI pipelines, since our tests are adding the indirect dependency.

@bcmills PTAL and ask for more info if needed.

P.D.: I've omitted the .go files in the example containing just underscore-imports to make tidy keep the imports, and the go.sum files.

[bcmills]

@colega, this was fixed as #26902 and will not be backported to the 1.12 branch. If you can still reproduce it (or something similar) using go1.13beta1, please open a new issue.