cmd/go: checksum mismatch for x8s.io/[email protected]+incompatible

[jharshman]

What version of Go are you using (go version)?

Go 1.11

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

darwin
amd64

What did you do?

go build ./...
reproduction is difficult because including the package k8s.io/client-go causes incompatible dependencies to be pulled in which is something I have to solve manually.

What did you expect to see?

A repeat of successful builds from the day before.

What did you see instead?

A checksum mismatch this morning after cloning the project to my personal laptop and attempting a build.

go: verifying k8s.io/[email protected]+incompatible: checksum mismatch
        downloaded: h1:7Zl+OVXn0bobcsi4NEZGdoQDTE9ij1zPMfM21+yqQsM=
        go.sum:     h1:2pUaSg2x6iEHr8cia6zmWhoCXG1EDG9TCx9s//Aq7HY=

[jharshman]

Okay so the workaround is to manually go get the correct version and then run the build again. This is a little unpredictable IMO. I'm guessing this stuff will clear up as more packages add module support and semantic versioning.

[katiehockman]

@jharshman is there further investigation that needs to be done for this issue? Is there documentation that you are suggesting should be updated? If not, it sounds like this issue can be closed.

[bcmills]

+incompatible versions should not cause checksum mismatches.

Is it possible that you were running more than one build concurrently? (#26794)

[jharshman]

@katiehockman I would say this particular issue shouldn't be closed because this seems to be a symptom of a larger issue and needs to be addressed. As @bcmills mentioned, +incompatible versions should not cause checksum mismatches but in this case it has.

And @bcmills to answer your question, no this was the only build running. There were no other builds going on concurrently.

[F21]

Also seeing this problem but not sure if this is due to GOOS mismatch. I am using WSL (Ubuntu x64) on Windows 10 to run go mod tidy and generate the corresponding go.sum file. I am building and testing using Gitlab CI using a Go 1.11 Alpine x64 container and I am seeing:

go: verifying k8s.io/[email protected]+incompatible: checksum mismatch
	downloaded: h1:7Zl+OVXn0bobcsi4NEZGdoQDTE9ij1zPMfM21+yqQsM=
	go.sum:     h1:2pUaSg2x6iEHr8cia6zmWhoCXG1EDG9TCx9s//Aq7HY=

go version on both the ci machine and my own machine outputs: go version go1.11 linux/amd64.

I also recreate the go.sum and go.mod files using an Go 1.11 alpine container (the same one used in CI) and still see the same problem.

[jharshman]

@katiehockman please remove the WaitingForInfo label unless there is more information that you are requesting.
@bcmills is there a solution in the pipe for this? It's very reproducible so I can image other users are experiencing it as well.

The best solution I have found is to go mod vendor once you have a working build.

[bcmills]

@jharshman Could you try a Go build from source at commit 1bca6ce and see if it still reproduces?

I wonder if this is a side-effect of #27153.

[leakingtapan]

Im having similar issue with golang master branch on my travis build. What's resolution for this?

[bcmills]

@leakingtapan, try a go built from source at the above-mentioned commit, and let us know whether it helps.

For CI-related issues, though, you're more likely running up against #26794.

[leakingtapan]

I tried 1bca6ce with no luck. I also tried 8081862 which was working on my Travis build, but now even this is failing.

go: verifying k8s.io/[email protected]: checksum mismatch
        downloaded: h1:AAwZncqPvpeV6DPF8/z2g05/79cumA6IunGDByI1bOs=
        go.sum:     h1:2/lmzYbN17Mr23mX/p4ODMm/MBWTeu2Q1Bvsw82YC40=

[ldb]

Same issue here:

go: downloading k8s.io/client-go v9.0.0+incompatible
go: downloading k8s.io/api v0.0.0-20181006043042-ded30614edd7
go: verifying k8s.io/[email protected]+incompatible: checksum mismatch
        downloaded: h1:/PdJjifJTjMFe0G4ESclZDcwF1+bFePTJ2xf+MXjcvs=
        go.sum:     h1:NXWpDuPFeVB5lYP1fTqJUtwigjtmRXJNtndnN53ldGI=

[jasonkeene]

@leakingtapan @Cosmonawt @jharshman @F21

If you are running 1bca6ce your h1 hashes will, unfortunately, have to be updated once. Both kubernetes/kubernetes and kubernetes/client-go use export-subst which is disabled by this change:

• https://github.com/kubernetes/kubernetes/blob/7f23a743e8c23ac6489340bbb34fa6f1d392db9d/pkg/version/base.go#L58
• https://github.com/kubernetes/client-go/blob/03bfb9bdcfe5482795b999f39ca3ed9ad42ce5bb/pkg/version/base.go#L58

If you are running 1bca6ce you should get the same hash moving forward. Just delete the old hash and use the new one. Again, you should only have to do this once.

I imagine a lot of folks will run into this issue since k8s repos appear to use export-subst in various places.

If you are seeing inconsistent hashing results I recommend:

1. Build go from source after 1bca6ce
2. Delete the old h1 hash for k8s.io/kubernetes k8s.io/client-go and any other repo using export-subst.
3. Build your project and record the new hash for k8s.io/kubernetes k8s.io/client-go etc

If the hash changes for you after that then please report back.

[leakingtapan]

@jasonkeene thx for providing the solution. In order to not break people who is using go1.11 to build our package, I decided to stay with go1.11 for a while. This makes my travis build and mac build happy. However, when I tried build on a linux machine the issue even happen on go1.11, which is surprising. Here is the output:

mac

>> go version
go version go1.11 darwin/amd64

>> echo $GO111MODULE
on

>> make
make
mkdir -p bin
go build -o bin/aws-ebs-csi-driver ./cmd/aws-ebs-csi-driver

linux

>> go version
go version go1.11 linux/amd64

>> echo $GO111MODULE                                                                                                                                                                                                            
on

>> make
mkdir -p bin
go build -o bin/aws-ebs-csi-driver ./cmd/aws-ebs-csi-driver
go: verifying k8s.io/[email protected]: checksum mismatch
        downloaded: h1:AAwZncqPvpeV6DPF8/z2g05/79cumA6IunGDByI1bOs=
        go.sum:     h1:2/lmzYbN17Mr23mX/p4ODMm/MBWTeu2Q1Bvsw82YC40=
make: *** [aws-ebs-csi-driver] Error 1

[jasonkeene]

Yes this might be the problem 1bca6ce was intended to solve. Can you diff the contents of the two zip files that produce the different hashes?

[jharshman]

Thanks for the solution @jasonkeene , but building Go from source is the verification, not a long term solution. We need to be able to build projects reliably cross teams, platforms, and in CI.

It sounds like a bug-fix release needs to be cut from this.

[jasonkeene]

@jharshman For sure compiling from source was just to test to make sure this was your issue.

As far as the bug fix release, pretty sure @bcmills is on it here: #28094

I would still like you to try the above steps to make sure you are not running into another issue and that the bug fix will actually fix your problem.

[jharshman]

@jasonkeene sure let me verify that I don't hit any other issues 👍

[jsternberg]

I'm not sure if it's related, but this may be related to an issue we've been having with modules: #27952

We've had situations where the incomplete project didn't compile, but we've also had times that it compiled, but had the wrong hash because files were missing.

[jharshman]

@jasonkeene verified the fix by compiling from source after 1bca6ce. Works after removing the older hashes.

[jasonkeene]

Awesome, good to know!

[simonpasquier]

@lastzero you're most probably hitting #29278 (comment)

[lastzero]

@simonpasquier Thank you! Looks like the follow-up "bug". For those also affected: I had to change settings in GoLand (didn't automatically detect latest Go version) plus delete Go module caches locally and on Travis CI.

[irvifa]

Using 1.11.5 and still have this impact

OS: Darwin/Mojave

Edit: apparently it works after I do:

rm go.sum
go mod vendor

[int128]

If your machine is macOS or WSL and your CI is Linux, you can add this command to the CI config.

sed -e '/^k8s.io\/client-go /d' -i go.sum

Here is an example for CircleCI:

version: 2
jobs:
  build:
    docker:
      - image: circleci/golang:1.11.2
    steps:
      - checkout
      # workaround for https://github.com/golang/go/issues/27925
      - run: sed -e '/^k8s.io\/client-go /d' -i go.sum
      - run: go vet

[mei-rune]

d:\xxx>go version
go version go1.12.4 windows/amd64

d:\xxx>go mod tidy
go: downloading github.com/kevinburke/go-bindata v3.11.0+incompatible
verifying github.com/kevinburke/[email protected]+incompatible: checksum mismatch
downloaded: h1:RcC+GJNmrBHbGaOpQ9MBD8z22rdzlIm0esDRDkyxd4s=
go.sum: h1:GiPs9jxaG2xY1B5Dt/d/yHUOMlTk14uS35VcmHrdo4I=

[mei-rune]

verifying github.com/kylelemons/[email protected]: che
cksum mismatch
downloaded: h1:mkl3tvPHIuPaWsLtmHTybJeoVEW7cbePK73Ir8VtruA=
go.sum: h1:SzHLOWB819X9aUXh2ct6krDIoIujoTsj5J1rDMjkRyo=

[lastzero]

@runner-mei Did you try go clean -modcache?

[mei-rune]

@lastzero oooo, it is ok!