proposal: cmd/go: ignore +incompatible versions as of Go 1.14 #34217
Comments
ProposalI propose that, if the main module's
|
Is this saying that if you are building module foo, which requires module bar, and bar has a go.mod containing If so, does that mean the version of baz selected for the build could change (that is, ending up with some later commit than the commit tagged And if so, what is the brief rationale for that? |
It would be treated as if module In that case, your module ( That means that you could indeed end up with a version of |
|
The rationale is that I don't want to make the MVS step use any notion of “version precedence” that isn't ordinary semantic versioning. (I especially don't want to have to walk back from |
But would that then mean go.mod is not providing reproducible builds if the versions selected in the build change upon upgrading to Go 1.14? If so, would that be reasonable to do given prior statements around future releases being able to properly consume (Sorry for the multiple questions; mainly still trying to see if I understand the proposal). |
It would mean that the transition from Go 1.13 to Go 1.14 would not reproduce the same set of dependencies, but since the entire toolchain and standard library changes at each major release, that boundary is not exactly “reproducible” to begin with. Once you run a The more interesting problem occurs when going back from 1.14 to 1.13, since the version selected (and written to the |
This comment has been minimized.
This comment has been minimized.
I haven't fully considered this yet, but it seems like a significant, breaking change in MVS. This would change builds in a more significant way than toolchain and standard library changes do (those tend to be in Hyrum's law territory). An obvious consequence: suppose you depend on module A (which has migrated to modules), which depends on module B (which has not) at v2.0.0+incompatible. With this change, the B requirement is ignored, and we get a pseudo-version based on v3.0.0, the newest major version of B without a go.mod file. The B API has been rewritten between major versions, and the upgrade breaks the build. The author of the main module now needs to either (of course this can already happen if someone else requires B v3.0.0+incompatible) |
|
@jayconrod, that's an interesting consideration (and somewhat ties in to #34165). One alternative would be to only apply this proposal if the main module's That has the advantage of retaining compatibility with as much of the It also has the disadvantage of changing the meaning of a module's requirements depending on which other module is importing it, which makes fixes and patches more difficult to test — but perhaps the advantage of clearly attributing the change outweighs that disadvantage, and at any rate the main module can already change the meaning of a module's requirements using |
|
Thinking about it some more, gating on |
|
If we're gating on One problem with this: adding |
|
@jayconrod, I think we could reasonably treat a At the very least, I think we would need to give the main module the opportunity to to |
|
This seems plausible but I haven't thought through it all carefully. |
|
CC @Helcaraxan |
|
@bcmills To help people confirm their understanding of the proposal, what do you think about giving a couple examples of final pseudo-versions? For example, for tags like Is it correct that the final result would translate to a fourth form of pseudo-version being defined, rather than the current three forms:
Or if not a forth form, then maybe it is just a tweak to the current three definitions, but a VCS tag like Separately, as I understand it, part of the overall intent for the proposal is to help simplify the cmd/go code (cutting down on corner cases and corresponding bugs) as well as making things simpler for users. But if cmd/go will need to forever understand old go.mod files, is there a large gain in simplicity for the cmd/go code? (I can imagine there could be simplicity gains if only reading old modules needs to be supported, but at least wanted to ask the question).
Somewhat related to prior comment, perhaps this could get rolled out across two releases -- first, in say Go 1.14, just rewrite Also, if these are new pseudo-versions, is there an opportunity to inject a suggestive word into the pseudo-version itself, so that people have a higher chances of (a) understanding when first encountering these (without ever having read the specific piece of doc that describes them), or at least (b) recognizing them when they see them again later? Or maybe injecting a comment? Finally, there is some subtlety in the discussion (including it is slightly tricky in some of the discussion to be 100% sure which "major version" is being mentioned), so sorry if some of these questions are off base, or if I have otherwise misunderstood. |
|
@bcmills Also, I think I did not fully understand the last portion of this comment (including whether or not it means Jay's immediately prior suggestion does not work, or if it is only tangentially related to Jay's comment):
(It seems like at least part of the issue might be related to how do you specify a |
It would be one of the usual forms. The resolution process is basically:
So, for example:
Or:
|
No; the benefit is mainly for users — particularly for maintainers with existing In particular, it would give maintainers a way to opt in without forcing their existing consumers to update import paths, assuming that either those consumers have already adapted to any breaking changes up to the last major version, or the maintainers are willing to go back to the API as it stood at the last |
|
In full disclosure, though, there are two possible simplicity gains in
|
A |
Metadata in the pseudo-version itself would not be backward-compatible to Go 1.13 and earlier, and so would prevent Go 1.13 users from consuming a 1.14 module. That's not out of the question, but seems a high cost in exchange for the marginal value of the metadata. We could probably inject a comment, at least. |
If the main module specifies |
|
Some more insights from a discussion with @jayconrod this morning. Ideally, we would like to do our best to respect The major problem with that approach is that it the version-to-pseudo-version mapping may be expensive to compute, so we don't want to recompute it every time we resolve dependencies. |
|
However, we could do the version-to-pseudo-version resolution once and record the result in the That would at least preserve the selected version at the 1.13-to-1.14 transition point, but wouldn't pick up |
|
One other alternative might be to cache the One option I'm considering for #26904 is to introduce a form of the We could similarly overload the replace github.com/Azure/go-autorest v13.0.1 => v1.1.2-0.20190906230412-69b4126ece6bOn the other hand, I don't see much point in overloading the remap github.com/Azure/go-autorest v13.0.1 => v1.1.2-0.20190906230412-69b4126ece6bA |
|
It seems to me that from a practical perspective the need to respect the In practice such breakages might not be too numerous, nor touch the majority of users, but it's yet another place where modules could potentially interfere with the "it just works" principle. The errors could also be very hard to decipher as they would arise in transitive and not direct dependencies of someone's project. Debugging this would then require a developer to learn in one swoop quite a bit about modules, dependency management subtleties, the actual implementation details of their dependencies and even the dependencies of their dependencies. This is not something that people would usually want to do. Hence if we can prevent it we should do so. On the more practical side I do like the idea of the The downside though of A potential middle ground could be to further make All this said... I do very much like the idea of removing the |
|
I thought about this a bit more, and I have some questions and concerns listed below. In general, I'm worried this is a significant change with subtle implications. I don't expect we're going to be able to foresee all potential issues, and I'd like to see if we can solve the problems listed above by doing something narrower and possibly also #24031.
|
|
I'd like to reiterate the previous commenters' support for keeping resolved versions the same. For any plan moving forward I think that, as a minimum requirement, modules should continue to build with exactly the same resolved dependencies on at least the last two versions of Go. People need to be able try out the latest version while still having the option to drop back to the previous version. So anything landing in 1.14 should change the |
|
Thanks @jayconrod for bring up the issues related to the module proxy.
Proxies that rely on the I feel adding this in 1.14 is too rushed to evaluate the impact and added complexity to the whole go ecosystem. |
A proxy may continue to advertise them, but may choose to omit them. (In general the contents of a proxy's I am not sure which direction we should recommend. As you note, newer versions of the
No. However, the equivalent non-
It may cause users of earlier
Agreed. And
|
I'll use that example as a starting point for a couple of questions, but let's also make up a v12.0.1 release that also does not have a go.mod. Let's pretend it was released a while ago (say, in 2018). That would give us:
Those sort properly as pseudo-versions. However, let's suppose tomorrow there is a new v12.0.2 release (given part of the point of having major versions is to allow patch releases for prior major versions). Let's say that gives us:
At that point, that no longer sorts properly compared to the pseudo-version generated for Let's take a second scenario. Let's pretend v12 is an import major release that they want to support for a long time, so they decide to adopt modules within the v12 train. Let's say that then gives us:
That also no longer sorts properly compared to the pseudo-version generated for Is that the way this would work? Again, sorry if any of this is off base. |
|
Sorry, my second scenario is not as relevant, given at that point it would be |
|
Correct me if I'm wrong but: One nice thing about this proposal is that converting Which means even if the user is not using Go 1.14, but the proxy he/she is using is internally using Go 1.14, then their proxy will properly tell them to update their go.mod file to the correct version. On the other hand, this can mean that if a proxy storage has one thousand +incompatible modules, then it will eventually have its own 1000 duplicates of the pseudo semver module. If we can run some off-go command to know how eaxctly a +incompatible translates to the proposed pseudo semver, then go proxy maintainers can write a script to clean up the incompatible modules once they have all the corresponding duplicates |
|
For the Go proxy we wrote at work the +incompatible situation has been a constant source of problems. Now that almost everyone is upgrading to go modules, I think we should simply require a go.mod file for all module imports, an drop all support for the +incompatible tag. This might create some transition problems, which can be dealt with a transition over the next 2 go releases, and using go1.14 or (go1.15) in the go.mod file, but it will make go modules and go module proxies a lot easier to implement. |
|
@beoran could you elaborate on
Do you mean that we should no longer support non-module imports? |
|
Yes. In hindsight, I think non module imports are more troublesome than they are useful. The convenience does not weigh up to the complexity. If we didn't have to support non module imports, or proxy would likely require 25% less code. |
|
That would, in my opinion, be a much too aggressive move that risks causing an insurmountable amount of pain on the entire Go ecosystem. Especially given that it is still to be decided whether modules are going to be the default flow in 1.14, as this might be pushed again to 1.15. It would also be kind-of a breaking change (although not formally as it's not part of the Go language spec). What such a change would mean is that projects will not be able to update their Given that many frequently-used projects have their oldest dependencies somewhere in the range of 2 to 7 years old that is not something that is a reasonable burden to put on these projects. Bear in mind that these dependencies are so old because they are no longer being developed and might simply have reached a stable and mature state 4+ years ago. This would mean that all such projects will need to be hard/soft forked (because the maintainer has likely moved on) or redeveloped from scratch. |
|
Also please note: this proposal is not going to make it into 1.14 and will likely be reworked in the form of a new proposal by @bcmills, based on the golang-tools call earlier this week. |
|
Yeah, I'm going to go ahead and withdraw this proposal — the consensus seems to be that it's too invasive. I'll file another one for something less invasive. |
We do not want to contribute to informing Google of every single user that uses go-libp2p, thanks. Also, the default proxy (proxy.golang.org) contains old and deprecated `+incompatible` versions that the Go toolchain selects over the more recent go-modded versions. See golang/go#34189 and golang/go#34217.
We do not want to contribute to informing Google of every single user that uses go-libp2p, thanks. Also, the default proxy (proxy.golang.org) contains old and deprecated `+incompatible` versions that the Go toolchain selects over the more recent go-modded versions. See golang/go#34189 and golang/go#34217.
Abstract
This is a proposal to ignore
+incompatibleversions found in the module graph starting with Go version 1.14.Background
The
gocommand requires that the import path of a module (or package within a module) match its semantically-versioned API. In particular, starting with major version 2 — the first breaking change from the API stabilized in major version 1 — the module path must end with a/vNsuffix indicating the major version of its API.However, prior to the introduction of modules, many Go package maintainers had already reused existing import paths (such as
github.com/google/go-github) across multiple major versions. To accommodate the migration to modules for users of those packages, thevgoprototype — and the initial module support released in thegocommand in Go 1.11 — allowed those existing major versions to be used directly, and even preferred them over compatible versions (#26238) under two conditions:The module with the incompatible version must not contain an explicit
go.modfile.The version must be annotated in the user's
go.modandgo.sumfiles with the suffix+incompatible, indicating that the selected version is not compatible with the original API for that path.Unfortunately, those exceptions introduce a number of problems:
proxy.golang.org: accidentally publishing a major version interplays poorly with Go modules #34189: If a user adopts modules (by adding an explicit
go.modfile) and then accidentally tags a pre-module commit with an inappropriate major version, the erroneous pre-module tag will always be semantically higher than the highest valid module-enabled release tag (1.99[…].99[…]).cmd/go: go modules ignores go.mod in semver repos not using semantic import versioning #27009: If a user has already tagged a major version above 1 and adopts modules (by adding an explicit
go.modfile), then they must either adopt the major-subdirectory layout for their project (disrupting developers on the project), or break upgrades for non-module users (who will be expecting the unversioned import path).cmd/go: creating v2+ modules has lower success rate than it could #31543: Since the damage of introducing breaking changes is already done, users often expect that their v2-or-higher repository can be migrated to modules without changing its import path, but for various reasons that does not work in the general case.
cmd/go: tries to download v2.2.2 (no suffix) when checking a replaced v2.2.2+incompatible for a missing import #33795, cmd/go: [email protected]+incompatible currently allows Semantic Import Versioning to be optional #32695: Since
+metadatatags in general are not semantically meaningful, the fact that+incompatibleis semantically meaningful requires numerous special cases that are difficult to test and maintain, and creates confusion about when incompatible versions are or are not allowed.cmd/go: 'go get' with semantic-version prefix doesn't fall back to matching tag or branch #29731: The possibility of using a
+incompatibleversion led us to support major-version wildcard queries, which are only useful for legacy repos and interfere with more useful branch queries.cmd/go: 'go mod download' fails with xyz+incompatible replacement targets while 'go mod verifies' works #34254: The constraints on
+incompatibleversions derive from the module path, but that introduces even more complexity (and makes things more difficult to debug) withreplacedirectives, which involve two module paths that may or may not impose the same constraints on the corresponding versions.In contrast, for another interesting case of legacy tagging — semantic versions with metadata (#31713) — we came up with what I believe is a simpler solution: instead of accepting the non-canonical version tags as-is, we instead rewrite them to canonical pseudo-versions with an appropriate major version.
In light of the problems we have encountered with incompatible major versions, I believe that we should have applied a similar strategy for incompatible versions: perhaps using them for “latest” version resolution, but rewriting them to canonical pseudo-versions.
Unfortunately, the decision was made, and cannot be unmade in light of our subsequent experience without breaking compatibility.
...or can it?
Observation
Since a
+incompatibleversion cannot have an explicitgo.modfile, it cannot impose any transitive requirements on module selection. Therefore, a+incompatibleversion selected as the minimal version of a module cannot impact the version selected for any other module.This implies that if we ignore the
+incompatibleversions in the module graph entirely, we will not accidentally drop requirements that pertain to other modules.This leads to the following proposal (see the comment below; updates will be linked from here).
CC @jayconrod @thepudds @hyangah @katiehockman @heschik
The text was updated successfully, but these errors were encountered: