typing-extensions: version capping is now recommended?

[henryiii]

The current README now recommends version capping typing_extensions? See my post on this - it's a horrible idea, and for something like typing_extensions, I don't see any reason to recommend it. Almost every library trying to use typing needs to use typing extensions. It's generally important to require a recent version, as well, due to the nature of the development typing in Python. We are already dealing with the fallout of users (mostly Poetry) adding caps, but now all usages are supposed to be capped if they follow the readme? Recently black and tensorflow couldn't be installed in the same environment due to black requiring >=3.10 and tensorflow being capped to ~=3.7.0. Tensorflow how now removed their upper cap.

This forces two possibilities. Either typing extensions makes common releases (and at least yearly is promised by the dropping of Python versions == major version promise), which means we will have many unsolvable conflicts. Or typing-extensions never releases another major release (and has to keep Python 3.6 support forever).

For users, depending on typing-extensions will become a liability, causing solving errors, and no one will be able to depend on it anymore (and this is a 'backport' type library!), and instead have to add lots of TYPE_CHECKING checks and dummy classes.

Python has a flat dependency tree. It is not JavaScript.

including dropping support for older Python versions

This is pretty much literally the only thing not a breaking change by True SemVer. Requires-Python handles this. Everything else could be a breaking change via Hyrum's law. "Practical" SemVer, yes, but you don't cap for practical SemVer.

I would highly recommend avoiding assign users to cap. I'd also recommend deprecation periods.

[srittau]

So what's your solution? Not version capping will break projects. Often this will happen to the project's users, not the developers. That's not acceptable. There is a reason that version capping has been an industry standard since before SemVer was formally codified.

I see your point about not bumping the major version when we drop support for a Python version, though. We could probably drop that.

[henryiii]

Please read the article above, and/or any of the linked articles.

You can't fix a version conflict as a user. You are completely dependent on some developer somewhere to fix this and release a new package. You can always fix a missing version cap as a user. This library targets developing libraries, so the next level of user is very often a library author.

Version capping is common in systems like JavaScript, where you have nested dependencies. Capping your dependency doesn't cause a version conflict (there are other issues with those systems, but not this one for this type of a library). It is also common in industry, where a company controls the software and can synchronize releases, etc. It is not a social construct in Python's Open Source community - Python itself, along with many of its libraries, have deprecation periods and deprecation warnings.

Solution: Don't recommend it, or clearly indicate you should only do it for applications. Ideally, add some deprecation policy. Maybe remove things from static typing a version before removing them from the library? And if someone really knows that they want a cap, they can add it themselves - describe the policy, and let them decide.

---

Example:

If you release typing_extensions 5 with 3.8 as the minimum required Python, and remove typing_extensions.Protocol, then anyone still supporting Python 3.8 is broken - but only if they dit not put an if on the typing_extension import. According to this, they should have pinned typing_extensions==4.*. But that immediately means they conflict with a library that requires typing_extensions>=5 - which libraries are supposed to be able to do, that's the point of a backport, to let users stuck on older versions use newer features!

But if they did the "right" thing and did:

if sys.version_info < (3, 8):
    from typing_extensions import Protocol
else:
    from typing import Protocol

They work just fine on typing_extensions 5 and didn't need to cap it!

Typing_extensions could have enforced this by not providing the backport if sys.version_info indicated it was in the standard library. And is it really important to remove the "wrapped" versions? It will break people, and SemVer is not something that helps here. Especially for a library like typing_extensions that everyone uses.

[srittau]

I find you argument unconvincing. Not version capping will break libraries as well. If a library turns out to work with a new major version of typing_extensions, this can be indicated in a new release by bumping the upper limit of the dependency. Until then using typing_extensions and that library together is unsafe. If you can live with the unsafety, use pip install --no-warn-conflicts. Otherwise, you need to wait with updating until all dependencies have upgraded.

It's unfortunate that Python doesn't support private dependencies or parallel install of different versions of a library, but that makes dependency capping even more important to avoid breaking software. Fortunately, Python doesn't have the JavaScript ecosystem problem of needing thousands of dependencies, yet. (Although there's a certain trend, not helped by the increasingly conservative view on adding new packages to Python's stdlib, or improving existing ones.)

[henryiii]

I'm not asking for a change in the library, I'm just asking for the explicit request for everyone to SemVer cap typing-extensions to be removed from the README. I think this should describe what the compatibility policy is, and leave it up to the reader if they want to cap - then it's their problem.

You are specifically recommending a practice I deem to be highly destructive to the Python ecosystem, and I think @brettcannon, @pganssle, @hynek, @layday, @gaborbernat, and @pradyunsg dislike the practice too. Avoiding the recommendation keeps this from being opinionated.

Note this is practice is very new to the Python ecosystem. Most packages did not cap historically before Poetry. Typing extensions didn't cap. This is being added then talked about like it was a multi-year old practice. Now it's becoming more common, and many packages have been forced to remove capping due to solver conflicts, like TensorFlow. Even temporary capping has affected packaging and others.

[henryiii]

I find you argument unconvincing

My example above or the 10,000 word article, along with the linked ones from Brett, Hynek, and Bernát?

pip install --no-warn-conflicts

You are assuming an end user, not a library. There are massive numbers of libraries using this, and typing-extensions is a dependency, not something you are (always) pip installing. And there are lots of other installers and systems, like Poetry, PDM, Pipenv, Hatch, etc.

[JelleZijlstra]

I agree that we shouldn't recommend capping. For typing-extensions specifically, it's rather unlikely that we'll make breaking changes (we can easily just re-export things from typing forever), and it's very common for users to want to require a new version because they want to use features from a new PEP. (Henry's linked post explicitly gives typing-extensions as an example of this case.)

[srittau]

I agree that we should avoid making breaking changes. One more reason to recommend capping, because when we do it's more likely to be for a big incompatible change.

[brettcannon]

I agree that we should avoid making breaking changes. One more reason to recommend capping, because when we do it's more likely to be for a big incompatible change.

But your definition of "big" might not meet my definition, and yet by doing a cap I don't get to make that decision for myself and instead pass that decision-making to you. This can become a resolution conflict in an installer because if I cap to e.g. <4 as recommended even though I'm compatible with v4 -- I can't necessarily know that until the v4 is released and I test -- and another package requires v4, the installation will fail. So by assuming in the future my package will fail instead of testing that compatibility you end up potentially capping the entire environment or leading to an installation failure. And so the cap against an unknown future guarantees work in the future to change the cap (or being dropped due to version incompatibilities with other packages that are deemed more important), while not capping means there's only the potential of work.

[srittau]

Capping is safe, not capping is unsafe, introducing potentially quite subtle bugs. Not capping means breaking software. I'm not willing to trade the convenience of potentially upgrading some software a bit earlier for the uncertainty of breaking software. Especially not in a package that is all about safety at the expense of convenience.

[gaborbernat]

I'm not sure how you use this package. I use it mainly to use typing features not yet available on my target interpreter version. As long as a new version does not introduce incompatible syntax there's no way a new version breaks my project. What does tend to cause issues though, is some libraries requiring newer versions of this package then some other libraries allow. Hence why I personally prefer not to pin. Now on the other hand for mypy even minor patch version break the world! This is because the type chequer got smarter and now it detects more typing issues. So for type checking I always pin down to the exact version. In that world, upper capping offers me no advantages. People put more trust in version caps than they should, and on average upper caps cause more issues than worth.

[srittau]

I am closing this issue now due to the brigading by OP, which won't allow an honest debate.

[hauntsaninja]

While OP did tag a lot of people, OP's opinion is valuable and I respect the opinions of the people he tagged, particularly so on matters of packaging. I do not wish for OP to feel unwelcome in raising this issue. I too have suffered at the hands of preemptive capping and think it's bad.

That said, this discussion seems to have stopped being productive. Since I can't think of a situation why typing_extensions would ever make a major release, arguments here seem somewhat hypothetical... and I'd rather avoid fights over hypotheticals.

I would still be interested if people can think of situations where we would make a major release. We could then talk about that situation constructively, which might help ground discussion. A good example of this is "major release for dropping old Pythons", which as a result of this issue, we won't do :-)

[pradyunsg]

Well, this is going to be my only comment here, since spirits here seem high. I do think this was closed extremely prematurely, in a manner intended to shut down discussion, but that's merely my opinion. The blog posts linked in the OP go into great detail on this topic (and have TL;DRs) and I don't see any acknolwdgement that the maintainers have actually read it. I'll encourage folks to do so, if they haven't already.

arguments here seem somewhat hypothetical

Capping is inherently a hypothetical practice -- you're operating under the hypothesis that whatever change is made in a new major release, it will be a breaking change for everyone, that you'd need to protect them from. Each of the terms in italics are individual assumptions that you're enforcing on all users of a package that does such pins.

I would still be interested if people can think of situations where we would make a major release.

If you'll never cut a major release, then why bother recommending users to cap things? :)

IIUC, the motivation here is that this practice will cause a lot of pain, whenever you cut a major release in the future. Python is not Javascript or Rust. You can't simulataniously install two versions of the same package in a Python environment, and you're explicitly telling users to do something that will cause pain in the ecosystem whenever you release a new major version. Details of how and why live in the blog posts linked in OP.

[srittau]

I've locked the conversation now, since the pile-on caused by the brigading doesn't seem to stop. I'm aware that most people commenting here are probably still able to comment, despite this, but I hope that you still honor the lock.