Adding protocols and aliases from _typeshed to typing-extensions

[srittau]

This was already partly discussed in #9. I think we should start adding some commonly used protocols, type aliases, and other features from _typeshed to typing_extensions so they are available at runtime. This would prevent awkward guarded imports at runtime.

I would suggest to put this into a submodule to separate those types from the "backports and experimental features" that are added to the main namespace. Suggestion: typing_extensions.ext. For a start I would suggest to add the following:

• Some simple dunder protocols: SupportsNext, SupportsAnext, SupportsTrunc, SupportsDivMod, SupportsRDivMod. (I would hold off on protocols like SupportsAdd for now as they are using Any.)
• SupportsGetItem, SupportsItems and SupportsKeysAndGetItem
• The path aliases StrPath, BytesPath. StrOrBytesPath probably doesn't have much value over StrPath | BytesPath.
• Some simple I/O protocols: SupportsRead, SupportsReadline, SupportsWrite

[srittau]

I will prepare a PR once python/typing#1182 is implemented.

[hauntsaninja]

Thanks for pushing this forward! It's great to have a place where we can put generally useful definitions.

That said, there are a couple things it might be nice to have clarity on before we hit merge on #45

First, I'm curious what the relationship we expect between utils and typing.py is. Do we expect to ever merge definitions into typing.py? Note that if we do, the currently clear line between backport and utils gets murky. (If the answer is no, then I'm definitely in favour of subpackage. If the answer is yes, then I have some more questions about when we'd do so)

Second, it might be worth discussing and documenting criteria for inclusion and for stability. As it stands, I think we'll need to have conservative criteria because typing_extensions is so totally insanely widely used. For instance, if we were to make a breaking change that causes many libraries to pin typing_extensions, we'd limit the whole ecosystem's ability to access new type system features (related, #40 )!!

Note that the chances of us making breaking changes are very real. An example of this is the issue linked above, #9, which sparked a lot of interest in doing this in the first place — but SupportsLessThan has since been removed from typeshed.

Third, I'm a little concerned about type checker special casing of typing_extensions and versioning. I'm worried that utils just has a totally different set of pros and cons when it comes to where the types should come from, i.e. whether implicitly through type checker vendoring or explicitly through PEP 561. See python/typeshed#7912

Right now we want the backport typing_extensions stub to come from the type checker, since a) they're tightly coupled to type checker versions, b) we need the stdlib to be able to depend on magically special cased typing_extensions objects, and c) because it allows type checkers to work without typing_extensions being installed or with a "wrong" version installed (--python-executable, pyright). But I think these considerations apply much less to utils since they're non-magical; it feels like PEP 561 types could overall work better for utils, particularly if we regularly add stuff here.

(Having written the third bullet, I'm starting to feel more warmly towards using a separate PEP 561 package than I was before. It might be nice to be able to be a lot more aggressive about inclusions and breaking changes without sweating over whether people actually use divmod or whether SupportsAdd has an Any we might want to get rid of later. And because of PEP 561, users will be able to version things themselves and be alerted to runtime mismatches or any changes we make, without being linked to their type checker version)

[hmc-cs-mdrissi]

I'm a bit unsure if it's better to discuss here or in pr, so my comments touch topics mentioned in both.

On naming the one non utils name I see is aliases.py/protocols.py. Maybe type_definitions.py if you want to keep them all together in one file. I have a couple helper files like this and tend to call them something like type_aliases.py/types.py as they mostly have a collection of aliases.

Protocol I'd most like is SupportsRichComparison for various builtins that rely on it. And I agree that criteria for inclusion of protocols/aliases would be very helpful.

I also either lean that type extensions and _typeshed useful types should be independent packages or any types included would need to be very stable. I have dealt with a number of packages with upper bounds like typing extensions <4.0 that led to a number of hacks (mostly if TYPE_CHECKING and care for runtime inspection) to use certain typing features. Some of them I made prs to fix bound, but releases are occasional and recently I eventually gave up on 2 packages and forked them to change there dependency constraints. I wish there was an easier way to override dependency conflicts.

Part of my preference for a separate pypi package is that number of people that'll have typing extensions as a transitive dependency is much higher then number that would have a separate _typeshed package as a dependency.

[Avasam]

I'd like to request types.NoneType / _typeshed.NoneType
use for this came up in https://bugreports.qt.io/browse/PYSIDE-2284

[hauntsaninja]

Good timing, I started work on a separate PEP 561 package (planning on calling it useful_types). If I get enough time, I'll push something this weekend :-)

[JelleZijlstra]

I'm coming around to the opinion that it's better to make a separate package for "useful types". As @hauntsaninja discussed above, typing-extensions is treated very specially in the typing ecosystem, in that typecheckers treat it as part of the standard library. Because of that special status, it's good to remain strict about what to include in typing-extensions: only features from CPython and from pending PEPs.

I would welcome a standalone package to distribute useful reusable types, though. Any that turn out to be especially useful could be added to CPython and then here.

[JelleZijlstra]

Closing this following the policy to only include things that are in CPython or in a PEP. Let's instead create a separate useful-types package.

[hauntsaninja]

https://pypi.org/project/useful-types/ now exists!