Cannot get latest version: module contains a go.mod file, so module path should be github.com/pingcap/tidb/v3

[KateGo520]

Background

The github.com/pingcap/tidb uses Go modules and the current release version is v3. And it’s module path is "github.com/pingcap/tidb", instead of "github.com/pingcap/tidb/v3". It must comply with the specification of "Releasing Modules for v2 or higher" available in the Modules documentation. Quoting the specification:

A package that has opted in to modules must include the major version in the import path to import any v2+ modules
To preserve import compatibility, the go command requires that modules with major version v2 or later use a module path with that major version as the final element. For example, version v2.0.0 of example.com/m must instead use module path example.com/m/v2.
https://github.com/golang/go/wiki/Modules#releasing-modules-v2-or-higher

Steps to Reproduce

GO111MODULE=on, run go get targeting any version >= v2.1.0 of the pingcap/tidb:

$ go get github.com/pingcap/[email protected]
go: finding github.com/pingcap/tidb v3.0.12
go: finding github.com/pingcap/tidb v3.0.12
go get github.com/pingcap/[email protected]: github.com/pingcap/[email protected]: invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v3

run go get github.com/pingcap/tidb, the version will stuck in v2.0.11:

$go get github.com/pingcap/tidb
go: finding github.com/pingcap/tidb v2.0.11+incompatible
go: downloading github.com/pingcap/tidb v2.0.11+incompatible
go: extracting github.com/pingcap/tidb v2.0.11+incompatible

SO anyone using Go modules will not be able to easily use any newer version of pingcap/tidb.

Solution

1. Kill the go.mod files, rolling back to GOPATH.

This would push them back to not being managed by Go modules (instead of incorrectly using Go modules).
Ensure compatibility for downstream module-aware projects and module-unaware projects projects

I see these dependencies in your go.mod file, which need modle awareness. So you'd better not use third-party tools(such as: Dep, glide, govendor…).

github.com/Jeffail/gabs/v2 v2.5.1 
github.com/pingcap/pd/v4 v4.0.0-rc.2.0.20200730093003-dc8c75cf7ca0 

You also need to update the import path to:

import github.com/Jeffail/gabs/…
import github.com/pingcap/pd/…

2. Fix module path to strictly follow SIV rules.

Patch the go.mod file to declare the module path as github.com/pingcap/tidb/v3 as per the specs. And adjust all internal imports.
The downstream projects might be negatively affected in their building if they are module-unaware (Go versions older than 1.9.7 and 1.10.3; Or use third-party dependency management tools, such as: Dep, glide,govendor…).

[*] You can see who will be affected here: [85 module-unaware users, i.e., GayStudio/openbilibili, GDWGH/mu-li-xian-bei, smallnest/soar]
https://github.com/search?o=desc&q=%22github.com%2Fpingcap%2Ftidb%22+filename%3AGodeps.json+filename%3Avendor.conf+filename%3Avendor.json+filename%3Aglide.toml+filename%3AGodep.toml&s=indexed&type=Code

If you don't want to break the above repos. This method can provides better backwards-compatibility.
Release a v2 or higher module through the major subdirectory strategy: Create a new v3 subdirectory (github.com/pingcap/tidb/v3) and place a new go.mod file in that subdirectory. The module path must end with /v3. Copy or move the code into the v3 subdirectory. Update import statements within the module to also use /v3 (import "github.com/pingcap/tidb/v3/…"). Tag the release with v3.x.y.

3. Suggest your downstream module users use hash instead of a version tag.

If the standard rule of go modules conflicts with your development mode. Or not intended to be used as a library and does not make any guarantees about the API. So you can’t comply with the specification of "Releasing Modules for v2 or higher" available in the Modules documentation.
Regardless, since it's against one of the design choices of Go, it'll be a bit of a hack. Instead of go get github.com/pingcap/tidb@version-tag, module users need to use this following way to get the pingcap/tidb:
(1) Search for the tag you want (in browser)
(2) Get the commit hash for the tag you want
(3) Run go get github.com/pingcap/tidb@commit-hash
(4) Edit the go.mod file to put a comment about which version you actually used
This will make it difficult for module users to get and upgrade pingcap/tidb.

[*] You can see who will be affected here: [341 module users, e.g., ushell/yearning, chaos-mesh/go-sqlancer, chaos-mesh/horoscope]
https://github.com/search?q=pingcap%2Ftidb+filename%3Ago.mod&type=Code

Summary

You can make a choice to fix DM issues by balancing your own development schedules/mode against the affects on the downstream projects.

For this issue, Solution 2 can maximize your benefits and with minimal impacts to your downstream projects the ecosystem.

References

• https://github.com/golang/go/wiki/Modules#semantic-import-versioning
• https://golang.org/cmd/go/#hdr-Module_compatibility_and_semantic_versioning
• https://github.com/golang/go/wiki/Modules#releasing-modules-v2-or-higher

[KateGo520]

@crazycs520 @lzmhhh123

[zz-jason]

seems this issue is duplicated with #12135. @tiancaiamao PTAL

[tiancaiamao]

Some of our team member do not agree with this change.
So we haven't adopting the standard go semanic version way yet.
Hold on until we reach a consensus.

[yujunz]

So we haven't adopting the standard go semanic version way yet.

Does adding go.mod imply an opt-in?

[tiancaiamao]

To my knowledge, we do use go.mod, but not obey the standard rules.
And I'm also willing to change to github.com/pingcap/tidb/v3.

[yujunz]

To my knowledge, we do use go.mod, but not obey the standard rules.

It does not make sense, especially for a community project.

And I'm also willing to change to github.com/pingcap/tidb/v3.

What is other's justification of not obeying the standard rules?

[skyeskie]

Impact

Using go get (under go1.13 and go1.14) does not work for anything later than [email protected], which was the last version to not have modules.

Anyone using Go modules (and maybe some of the other package managers) will not be able to easily use any newer version of tidb

Reason

Go modules is implemented incorrectly.

In the description for modules, it heavily emphasizes that different major versions must have different import paths. That means not only updating the module description to module github.com/pingcap/tidb/v3 (or v2 or v4), but also imports within consuming code should look like:

import (
	"github.com/pingcap/tidb/v3/meta"
	"github.com/pingcap/tidb/v3/util"
	"github.com/pingcap/tidb/v3/util/expensivequery"
)

Options

Kill the go.mod files.

This would push them back to not being managed by Go modules (instead of incorrectly using Go modules). Importing specific versions would work, but they'd receive the +incompatible prefix and would need to be manually upgraded.

Probably best option short-term -- better to not use Go modules than use it incorrectly.

Also, best option for older branches/versions that aren't maintained.

Make breaking changes

Go ahead and change all the import paths as well.

This would probably only be acceptable on the v4 line, since it'd break any consumers.

Major version bump / repository change

Leave v4 as a dead version and bump to v5 with the path changes.

goforward / shims

The Go modules wiki page mentions

A more sophisticated approach here could exploit type aliases (introduced in Go 1.9) and forwarding shims between major versions residing in different subdirectories. This can provide additional compatibility and allow one major version to be implemented in terms of another major version but would entail more work for a module author. An in-progress tool to automate this is goforward.

Looking through the goforward code, it'd

1. Move all code to a new import path

2. Create a forward.go file for each package's public methods, types, interfaces, etc.

   Example for tidb/util/math package:

package hack

import (
      "github.com/pingcap/tidb/v3/util/math"
)

func Abs(n int64) int64 {
  return math.Abs(n)
}

You'd end up with all the code in a new path, either

• a new repo under say github.com/pingcap/tidb-mod
• a new branch in the same repo (bump to 2.3, 3.1, 4.1, etc)
• use the folders structure on mainline branch (see Major Subdirectory structure on the Modules page).

Then development would take place in the new branch/repository, with the existing only updated with tags and/or with new things to forward.

[tiancaiamao]

PTAL @siddontang

[lonng]

This is a trade-off of the release management and project development, I think the standard rule of go modules conflicts with our development mode, there will be some extra works when we want to cherry-pick a bugfix to release-2.x/3.x/4.x(maybe more release branches). You can edit the go.mod directly to resolve the problem.

I apologize for the inconvenience. I think this is mainly a design flaw of Golang modules.

[yujunz]

Kill the go.mod files.

This would push them back to not being managed by Go modules (instead of incorrectly using Go modules). Importing specific versions would work, but they'd receive the +incompatible prefix and would need to be manually upgraded.

Probably best option short-term -- better to not use Go modules than use it incorrectly.

Also, best option for older branches/versions that aren't maintained.

This sounds a reasonable option to me. It is fine for user to receive +incompatible prefix and would need to be manually upgraded. It is much better than appearing to be compatible (opt-in with go.mod) while actually not working for downstream projects at all.

[skyeskie]

This is a trade-off of the release management and project development, I think the standard rule of go modules conflicts with our development mode, ...
I apologize for the inconvenience. I think this is mainly a design flaw of Golang modules.

Go is a rather opinionated language - as is the modules portion. The modules documentation states using proper semver and having different import paths is a requirement to use Go modules.

Additionally, they document the semver requirement was done so they could avoid the diamond dependency problem (explained at length https://research.swtch.com/vgo-import) and to allow gradually upgrading code (have different major versions at the same time).

---

there will be some extra works when we want to cherry-pick a bugfix to release-2.x/3.x/4.x(maybe more release branches).

Cherry-picks would still work if they don't change the import section (for internal packages).

Arguably if the imports change, it's more likely for there to be differences between major versions where simply cherry-picking wouldn't be appropriate.

---

You can edit the go.mod directly to resolve the problem.

You'd have to make sure that it references the specific commit and not the actual version number. Otherwise, it'll break other commands like go list

It does look like you can currently use these commands for a workaround:

# go get github.com/pingcap/tidb@<branch>
go get github.com/pingcap/[email protected]

# go get github.com/pingcap/tidb@<commit hash>
go get github.com/pingcap/tidb@fcbcc4569fbd7aba9eb92092149f092346b934ec

Of course, using the branch name doesn't guarantee you get a working release.

Also, the 'version' it shows in the go.mod file ends up looking like this (for v2.1.19):

	github.com/pingcap/tidb v1.1.0-beta.0.20191220121606-fcbcc4569fbd

Regardless, since it's against one of the design choices of Go, it'll be a bit of a hack. Instead of go get github.com/pingcap/tidb@<tag>, the install procedure would be something like

1. Search for the tag you want (in browser)
2. Get the commit hash for the tag you want
3. Run go get github.com/pingcap/tidb@<commit hash>
4. Edit the go.mod file to put a comment about which version you actually used

---

I suppose if you want to keep using Go modules in developing, rather than straight-up delete the go.mod file rename it to ~go.mod. Then anyone developing TiDB could rename it back locally.

[tisonkun]

As my observation there is several voice talking about go submodule or be able to get tidb by go get. Remove the wontfix tag and revive the issue.

Reference

• https://internals.tidb.io/t/topic/364/7