From 4d9cc0adddf865125388d158359ef7fefeee0b09 Mon Sep 17 00:00:00 2001 From: Alice Purcell Date: Fri, 9 Jun 2023 16:31:47 +0100 Subject: [PATCH 01/13] Rewrite PEP 705 to focus on TypedDict changes --- peps/pep-0705.rst | 495 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 380 insertions(+), 115 deletions(-) diff --git a/peps/pep-0705.rst b/peps/pep-0705.rst index f62a8305c9b..a763ad2c232 100644 --- a/peps/pep-0705.rst +++ b/peps/pep-0705.rst @@ -1,5 +1,5 @@ PEP: 705 -Title: TypedMapping: Type Hints for Mappings with a Fixed Set of Keys +Title: TypedDict: Read-only and other keys Author: Alice Purcell Sponsor: Pablo Galindo Discussions-To: https://discuss.python.org/t/pep-705-typedmapping/24827 @@ -8,7 +8,7 @@ Type: Standards Track Topic: Typing Content-Type: text/x-rst Created: 07-Nov-2022 -Python-Version: 3.12 +Python-Version: 3.13 Post-History: `30-Sep-2022 `__, `02-Nov-2022 `__, `14-Mar-2023 `__, @@ -19,14 +19,19 @@ Abstract :pep:`589` defines the structural type :class:`~typing.TypedDict` for dictionaries with a fixed set of keys. As ``TypedDict`` is a mutable type, it is difficult to correctly annotate methods which accept read-only parameters in a way that doesn't prevent valid inputs. -This PEP proposes a type constructor ``typing.TypedMapping`` to support this use case. +As structural subtypes can add other keys in, it is also difficult for type-checkers to safely define covariant methods like ``update``, or support type narrowing. +This PEP proposes two new ``TypedDict`` flags, ``readonly`` and ``other_keys``, plus an associated type qualifier, ``typing.ReadOnly``. Motivation ========== Representing structured data using (potentially nested) dictionaries with string keys is a common pattern in Python programs. :pep:`589` allows these values to be type checked when the exact type is known up-front, but it is hard to write read-only code that accepts more specific variants: for instance, where fields may be subtypes or restrict a union of possible types. This is an especially common issue when writing APIs for services, which may support a wide range of input structures, and typically do not need to modify their input. -For illustration, we will try to add type hints to a function ``movie_string``:: + +Pure functions +-------------- + +Consider trying to add type hints to a function ``movie_string``:: def movie_string(movie: Movie) -> str: if movie.get("year") is None: @@ -77,106 +82,297 @@ The problem disappears if we don't have mutator methods in ``Movie``. This could This is very repetitive, easy to get wrong, and is still missing important method definitions like ``__contains__()`` and ``keys()``. +Updating nested dicts +--------------------- + +The structural typing of ``TypedDict`` is supposed to permit writing update functions that only constrain the types of entries they modify:: + + class HasTimestamp(TypedDict): + timestamp: float + + class Logs(TypedDict): + timestamp: float + loglines: list[str] + + def update_timestamp(d: HasTimestamp) -> None: + d["timestamp"] = now() + + def add_logline(logs: Logs, logline: str) -> None: + logs["loglines"].append(logline) + update_timestamp(logs) # Accepted by type checker + +However, this no longer works once you start nesting dictionaries:: + + class HasTimestampedMetadata(TypedDict): + metadata: HasTimestamp + + class UserAudit(TypedDict): + name: str + metadata: Logs + + def update_metadata_timestamp(d: HasTimestampedMetadata) -> None: + d["metadata"]["timestamp"] = now() + + def rename_user(d: UserAudit, name: str) -> None: + d["name"] = name + update_metadata_timestamp(d) # Type check error: "metadata" is not of type HasTimestamp + +This looks like an error, but is simply due to the (unwanted) ability to overwrite the ``metadata`` entry held by the ``HasTimestampedMetadata`` instance with a different ``HasTimestamp`` instance, that may no longer be a ``UserAudit`` instance. + +It is possible to work around this issue with generics (as of Python 3.11), but it is very complicated, requiring a type parameter for every nested dict. + + +Type discrimination +------------------- + +Another common idiom in JSON APIs is to discriminate between mutually exclusive choices with a single-entry dictionary, where the key on the dictionary distinguishes between choices, and constrains the associated value type:: + + class Movie(TypedDict): + name: str + director: str + + class Book(TypedDict): + name: str + author: str + + class EntertainmentMovie(TypedDict): + movie: Movie + + class EntertainmentBook(TypedDict): + book: Book + + Entertainment = EntertainmentMovie | EntertainmentBook + +Users of this pattern expect type-checkers to allow the following pattern:: + + def get_name(entertainment: Entertainment) -> str: + if "movie" in entertainment: + return entertainment["movie"]["name"] + elif "book" in entertainment: + return entertainment["book"]["name"] + else: + # Theoretically unreachable but common defensive coding + raise ValueError("Unexpected entertainment type") + +However, type-checkers will actually raise an error on this code; mypy, for instance, will complain that ``TypedDict "EntertainmentBook" has no key "movie"`` on the third line. This is because ``TypedDict`` does not prevent instances from having keys not specified in the type, and so the check ``"movie" in entertainment`` can return True for an ``EntertainmentBook``. + +Users can alternatively use a non-total ``TypedDict`` instead of a union:: + + class Entertainment(TypedDict, total=False): + movie: Movie + book: Book + +This ensures the ``get_name`` example type-checks correctly, but it no longer encodes the constraint that exactly one key must be present, meaning other valid code raises spurious type-check failures. In practice, we tend to see code using types like this either casting to the correct type, with the associated risk of mistakes, or moving the ``in`` checks to dedicated ``TypeGuard`` functions, reducing readability. + Rationale ========= -The proposed ``TypedMapping`` type allows a straightforward way of defining these types that should be familiar to existing users of ``TypedDict`` and support the cases exemplified above:: +The first two motivating examples can be solved by removing the ability to update one or more of the entries in a ``TypedDict``. This does not mean the entries are immutable; a reference to the underlying dictionary could still exist with a different but compatible type in which those entries have mutator operations. As such, these are not "final" entries; using this term would risk confusion with final attributes, which are fully immutable. These entries are "readonly". + +To support this, we propose adding a new boolean flag to ``TypedDict``, ``readonly``, which when set to True, removes all mutator operations from the type:: - from typing import NotRequired, TypedMapping + from typing import NotRequired, TypedDict - class Movie(TypedMapping): + class Movie(TypedDict, readonly=True): name: str - year: NotRequired[int | None] + director: str -In addition to those benefits, by flagging arguments of a function as ``TypedMapping``, it makes explicit not just to typecheckers but also to users that the function is not going to modify its inputs, which is usually a desirable property of a function interface. -Finally, this allows bringing the benefits of ``TypedDict`` to other mapping types that are unrelated to ``dict``. + class Book(TypedDict, readonly=True): + name: str + author: str + +In addition to these benefits, by flagging arguments of a function as read-only, it makes explicit not just to typecheckers but also to users that the function is not going to modify its inputs, which is usually a desireable property of a function interface. + +A new ``typing.ReadOnly`` type qualifier allows removing the ability to mutate individual entries, permitting a mixture of readonly and mutable entries. This is necessary for supporting the second motivating example, updating nested dicts:: + + class UserAudit(TypedDict): + name: str + metadata: ReadOnly[Logs] + +Finally, to support type discrimination, we add a second boolean flag to ``TypedDict``, ``other_keys``, which when set to True, prevents instances from holding any key not explicitly listed in the type:: + + class EntertainmentMovie(TypedDict, readonly=True, other_keys=False): + movie: Movie + + class EntertainmentBook(TypedDict, readonly=True, other_keys=False): + book: Book + + Entertainment = EntertainmentMovie | EntertainmentBook + + def get_name(entertainment: Entertainment) -> str: + if "movie" in entertainment: + return entertainment["movie"]["name"] + elif "book" in entertainment: + return entertainment["book"]["name"] + else: + raise ValueError("Unexpected entertainment type") Specification ============= -A ``TypedMapping`` type defines a protocol with the same methods as :class:`~collections.abc.Mapping`, but with value types determined per-key as with ``TypedDict``. +``TypedDict`` will gain two new boolean flags: ``other_keys`` and ``readonly``. A new ``typing.ReadOnly`` type qualifier is added. -Notable similarities to ``TypedDict``: +``other_keys`` flag +------------------- -* A ``TypedMapping`` protocol can be declared using class-based or alternative syntax. -* Keys must be strings. -* By default, all specified keys must be present in a ``TypedMapping`` instance. It is possible to override this by specifying totality, or by using ``NotRequired`` from :pep:`655`. -* Methods are not allowed in the declaration (though they may be inherited). +The ``other_keys`` flag to ``TypedDict``, when ``False``, indicates that instances may only contain keys explicitly listed in the type:: -Notable differences from ``TypedDict``: + class Album(TypedDict, other_keys=False): + name: str + year: int -* The runtime type of a ``TypedMapping`` object is not constrained to be a ``dict``. -* No mutator methods (``__setitem__``, ``__delitem__``, ``update``, etc.) will be generated. -* The ``|`` operator is not supported. -* A class definition defines a ``TypedMapping`` protocol if and only if ``TypedMapping`` appears directly in its class bases. -* Subclasses can narrow value types, in the same manner as other protocols. + class AlbumExtra(Album, TypedDict): + band: str # Runtime error -As with :pep:`589`, this PEP provides a sketch of how a type checker is expected to support type checking operations involving ``TypedMapping`` and ``TypedDict`` objects, but details are left to implementors. In particular, type compatibility should be based on structural compatibility. +This flag defaults to ``True``. +Type-checkers may rely on this restriction:: -Multiple inheritance and TypedDict ----------------------------------- + def album_keys(album: Album) -> Collection[Literal['name'] | Literal['year']]: + # Type checkers may permit this, but should error if Album did not specify `other_keys=False` + return album.keys() -A type that inherits from a ``TypedMapping`` protocol and from ``TypedDict`` (either directly or indirectly): +Type-checkers should prevent operations that would violate this restriction:: -* is the structural intersection of its parents, or invalid if no such intersection exists -* instances must be a dict subclass -* adds mutator methods only for fields it explicitly (re)declares + class AlbumExtra(TypedDict, other_keys=False): + name: str + year: int + band: str + + album: AlbumExtra = { "name": "Flood", year: 1990, band: "They Might Be Giants" } + album_keys(album) # Type check error: extra key 'band' + +``readonly`` flag +----------------- -For example:: +The ``readonly`` flag to ``TypedDict``, when ``True``, indicates that no mutator methods (``__setitem__``, ``__delitem``, ``update``, etc.) will be generated:: - class Movie(TypedMapping): + class NamedDict(TypedDict, readonly=True): name: str - year: int | None - class MovieRecord(Movie, TypedDict): - year: int + def get_name(d: NamedDict) -> str: + return d["name"] + + def set_name(d: NamedDict, name: str) -> None: + d["name"] = name # Type check error: cannot modify a read-only entry + +``typing.ReadOnly`` flag +------------------------ + +The ``typing.ReadOnly`` type qualifier is used to indicate that a variable declared in a ``TypedDict`` definition will not be included in any generated mutator method overloads:: + + from typing import ReadOnly + + class BandAndAlbum(TypedDict): + band: str + album: ReadOnly[Album] + +The ``readonly`` flag is equivalent to marking all entries as ``ReadOnly[]``, guaranteeing no entries are missed by mistake. To avoid potential confusion, it is an error to use both ``readonly=True`` and ``ReadOnly[]``:: + + class Band(TypedDict, readonly=True): + name: ReadOnly[str] # Runtime error: redundant ReadOnly qualifier + members: Collection[str] + +Alternative functional syntax +----------------------------- + +The :pep:`alternative functional syntax <589#alternative-syntax>` for TypedDict also supports these features:: + + EntityBand = TypedDict('EntityBand', {'band': Band}, readonly=True, other_keys=False) + BandAndAlbum = TypedDict(`BandAndAlbum', {'band': str, 'album': ReadOnly[Album]}) + +Interaction with other special types +------------------------------------ - movie: MovieRecord = { "name": "Blade Runner", - "year": 1982 } - - movie["year"] = 1985 # Fine; mutator methods added in definition - movie["name"] = "Terminator" # Type check error; "name" mutator not declared +``ReadOnly[]`` can be used with ``Required[]``, ``NotRequired[]`` and ``Annotated[]``, in any nesting order: -Inheriting, directly or indirectly, from both ``TypedDict`` and ``Protocol`` will continue to fail at runtime, and should continue to be rejected by type checkers. +:: + class Movie(TypedDict): + title: ReadOnly[Required[str]] # OK + year: ReadOnly[NotRequired[Annotated[int, ValueRange(-9999, 9999)]]] # OK + +:: + + class Movie(TypedDict): + title: Required[ReadOnly[str]] # OK + year: Annotated[NotRequired[ReadOnly[int]], ValueRange(-9999, 9999)] # OK + +This is consistent with the behavior introduced in :pep:`655`. -Multiple inheritance and Protocol ---------------------------------- +Inheritance +----------- -* A type that inherits from a ``TypedMapping`` protocol and from a ``Protocol`` protocol must satisfy the protocols defined by both, but is not itself a protocol unless it inherits directly from ``TypedMapping`` or ``Protocol``. -* A type that inherits from a ``TypedMapping`` protocol and from ``Protocol`` itself is configured as a ``Protocol``. Methods and properties may be defined; keys may not:: +To avoid potential confusion, it is an error to have a read-only type extend a non-read-only type:: - class A(Movie, Protocol): - # Declare a mutable property called 'year' - # This does not affect the dictionary key 'year' - year: str + class BandAlbumAndLabel(BandAndAlbum, readonly=True): # Runtime error + label: str -* A type that inherits from a ``Protocol`` protocol and from ``TypedMapping`` itself is configured as a ``TypedMapping``. Keys may be defined; methods and properties may not:: +It is valid to have a non-read-only type extend a read-only one. The subclass will not be read-only, but any keys not redeclared in the subclass will remain read-only:: - class B(A, TypedMapping): - # Declare a key 'year' - # This does not affect the property 'year' + class Album(NamedDict, TypedDict): year: int + album: Album = { name: "Flood", year: 1990 } + album["year"] = 1973 # OK + album["name"] = "Dark Side Of The Moon" # Type check error: "name" is read-only -Type consistency rules ----------------------- +Subclasses can redeclare read-only entries as non-read-only, allowing them to be mutated:: -Informally speaking, *type consistency* is a generalization of the is-subtype-of relation to support the ``Any`` type. It is defined more formally in :pep:`483`. This section introduces the new, non-trivial rules needed to support type consistency for ``TypedMapping`` types. + class Album(NamedDict, TypedDict): + name: str + year: int -First, any ``TypedMapping`` type is consistent with ``Mapping[str, object]``. -Second, a ``TypedMapping`` or ``TypedDict`` type ``A`` is consistent with ``TypedMapping`` ``B`` if ``A`` is structurally compatible with ``B``. This is true if and only if both of these conditions are satisfied: + album: Album = { name: "Flood", year: 1990 } + album["year"] = 1973 # OK + album["name"] = "Dark Side Of The Moon" # Also OK now -* For each key in ``A``, ``B`` has the corresponding key and the corresponding value type in ``B`` is consistent with the value type in ``A``. +Subclasses can narrow value types of read-only entries:: -* For each required key in ``A``, the corresponding key is required in ``B``. + class AlbumCollection(TypedDict, readonly=True): + albums: Collection[Album] + + class RecordShop(AlbumCollection, TypedDict): + name: str + albums: list[Album] + +Subclasses can also require keys that are read-only but not required in the superclass:: + + class OptionalName(TypedDict, readonly=True): + name: NotRequired[str] + + class Person(OptionalName, TypedDict): + name: Required[str] + + person: Person = {} # Type check error: "name" required + +Note that these are just consequences of structural typing, but they are highlighted here as the behavior now differs from the rules specified in :pep:`589`. + +Type consistency +---------------- + +A TypedDict type with ``other_keys=False`` is consistent with ``Mapping[str, V]``, where ``V`` is the union of all its value types. For instance, the following type is consistent with ``Mapping[str, int | str]``:: + + class Person(TypedDict, other_keys=False): + name: str + age: int + +A TypedDict type ``A`` is consistent with TypedDict ``B`` if ``A`` is structurally compatible with ``B``. This is true if and only if all of the following are satisfied: + +* For each key in ``B``, ``A`` has the corresponding key and the corresponding value type in ``A`` is consistent with the value type in ``B``, unless the key in ``B`` is of type ``ReadOnly[NonRequired[Any]]``, in which case it may be missing in ``A`` provided ``A`` allows other keys. +* For each non-read-only key in ``B``, the corresponding value type in ``B`` is also consistent with the corresponding value type in ``A``. +* For each required key in ``B``, the corresponding key is required in ``A``. +* For each non-read-only, non-required key in ``B``, the corresponding key is not required in ``A``. +* If ``B`` does not allow other keys, then ``A`` does not allow other keys. +* If ``B`` does not allow other keys, then for each key in ``A``, ``B`` has the corresponding key. Discussion: -* Value types behave covariantly, since ``TypedMapping`` objects have no mutator methods. This is similar to container types such as ``Mapping``, and different from relationships between two ``TypedDict`` types. Example:: +* All non-specified keys in a type that allows other keys are implicitly of type ``ReadOnly[NotRequired[Any]]``. + +* Read-only value types behave covariantly, as they cannot be mutated. This is similar to container types such as ``Mapping``, and different from non-read-only value types, which behave invariantly. Example:: - class A(TypedMapping): + class A(TypedDict, readonly=True): x: int | None class B(TypedDict): @@ -185,111 +381,180 @@ Discussion: def f(a: A) -> None: print(a['x'] or 0) - b: B = {'x': 0} + b: B = {'x': 1} f(b) # Accepted by type checker -* A ``TypedDict`` or ``TypedMapping`` type with a required key is consistent with a ``TypedMapping`` type where the same key is a non-required key, again unlike relationships between two ``TypedDict`` types. Example:: +* An TypedDict type ``A`` with no explicit key ``'x'`` that allows other keys is not consistent with a TypedDict type with a non-required key ``'x'``, since at runtime the key ``'x'`` could be present and have an incompatible type (which may not be visible through ``A`` due to structural subtyping). The only exception to this rule is if ``'x'`` is non-required, read-only and of type ``Any``. - class A(TypedMapping, total=False): - x: int +* A TypedDict type ``A`` with no key ``'x'`` that does not allow other keys may be consistent with a TypedDict type with a read-only, non-required key ``'x'``. Example:: - class B(TypedDict): + class A(TypedDict, total=False, readonly=True): + y: int + + class B(TypedDict, other_keys=False): x: int - def f(a: A) -> None: - print(a.get('x', 0)) + def f(a: A) -> int: + return a.get("y", 0) - b: B = {'x': 0} - f(b) # Accepted by type checker + def g(b: B) -> None: + b["x"] = f(b) # Accepted by type checker -* A ``TypedMapping`` type ``A`` with no key ``'x'`` is not consistent with a ``TypedMapping`` type with a non-required key ``'x'``, since at runtime the key ``'x'`` could be present and have an incompatible type (which may not be visible through ``A`` due to structural subtyping). This is the same as for ``TypedDict`` types. Example:: +Union Operation +--------------- - class A(TypedMapping, total=False): - x: int - y: int +The union operation creates a new dictionary with the merged keys and values of its two operands. As such, the result should be consistent with any type that can hold the possible key-value pairs, not just types compatible with the operand types. For example:: - class B(TypedMapping, total=False): + class A(TypedDict, readonly=True, other_keys=False): x: int - class C(TypedMapping, total=False): - x: int - y: str + class B(TypedDict, total=False, readonly=True, other_keys=False): + x: str - def f(a: A) -> None: - print(a.get('y') + 1) + class C(TypedDict): + x: int | str - def g(b: B) -> None: - f(b) # Type check error: 'B' incompatible with 'A' + def union_a_b(a: A, b: B) -> C: + # Accepted by type-checker, even though C is not read-only and + # allows other keys: + return a | b + +This is different from the usual compatibility rules, where the result of an operation has a defined type which the variable it is assigned to must be consistent with. A similar situation occurs with ``TypedDict`` and ``copy()`` or ``deepcopy()``. + +If the union of two TypedDict objects of type ``A`` and ``B`` are assigned to a TypedDict of type ``C``, the type checker should verify that: + +* if ``C`` does not allow other keys, neither ``A`` nor ``B`` allow other keys +* if ``C`` does not allow other keys, it contains all keys found in either ``A`` or ``B`` +* if a key ``'x'`` is found in ``A`` and ``C``, its type in ``A`` is consistent with its type in ``C``. +* if a key ``'x'`` is found in ``B`` and ``C``, its type in ``B`` is consistent with its type in ``C``. +* if a key ``'x'`` is required in ``C``, it is required in either ``A`` or ``B``. + +Notes: - c: C = {'x': 0, 'y': 'foo'} - g(c) # Runtime error: str + int +* The read-only status of the keys does not matter. A key can be read-only on just ``A``, just ``B``, or just ``C``, or any combination. +* A key found on ``A`` or ``B`` may be missed off ``C`` if it allows other keys. Type-checkers may however choose to flag this edge-case with a warning or error in some circumstances, if it is found to be a source of mistakes. -* A ``TypedMapping`` with all ``int`` values is not consistent with ``Mapping[str, int]``, since there may be additional non-``int`` values not visible through the type, due to structural subtyping. This mirrors ``TypedDict``. Example:: +Update Operations +----------------- - class A(TypedMapping): +Previously, ``clear()`` and ``popitem()`` were rejected by type checkers on TypedDict objects, as they could remove required keys, some of which may not be directly visible because of structural subtyping. However, these methods should be allowed on TypedDicts objects with all keys non-read-only and non-required and with no other keys allowed:: + + class A(TypedDict, total=False, other_keys=False): x: int + y: str + + a: A = { "x": 1, "y": "foo" } + a.popitem() # Accepted by type checker + a.clear() # Accepted by type checker + +``update`` has been difficult to type correctly due to the open nature of TypedDict objects. Keys not specified on the type could still be present (and constrained) due to structural subtyping, meaning type safety could be accidentally violated. For instance:: - class B(TypedMapping): + class B(TypedDict, total=False): x: int + + def update_b(b1: B, b2: B) -> None: + b1.update(b2) + + class C(B, TypedDict, total=False): + y: int + + class D(B, TypedDict, total=False): y: str - def sum_values(m: Mapping[str, int]) -> int: - return sum(m.values()) + c: C = { "x": 1, "y": 2 } + d: D = { "x": 3, "y": "foo" } + update_b(c, d) # c is no longer a C at runtime - def f(a: A) -> None: - sum_values(a) # Type check error: 'A' incompatible with Mapping[str, int] +Both mypy and pyright currectly permit this usage, however, as the only viable alternative has been to prevent calling ``update`` at all. - b: B = {'x': 0, 'y': 'foo'} - f(b) # Runtime error: int + str +With the addition of ``other_keys``, it becomes possible to more accurately type the update method: +* Declare a new read-only TypedDict type that does not allow other keys +* Copy all non-read-only entries to it +* Make all entries read-only and non-required +* Union this with an iterable of matching key-value pairs -Backwards Compatibility -======================= +For instance:: -This PEP changes the rules for how ``TypedDict`` behaves (allowing subclasses to -inherit from ``TypedMapping`` protocols in a way that changes the resulting -overloads), so code that inspects ``TypedDict`` types will have to change. This -is expected to mainly affect type-checkers. + class Example(TypedDict): + a: int + b: NonRequired[str] + c: ReadOnly[int] -The ``TypedMapping`` type will be added to the ``typing_extensions`` module, -enabling its use in older versions of Python. + class ExampleUpdateDict(TypedDict, total=False, readonly=True, other_keys=False): + a: int + b: str + # c is not present as it is read-only in Example + ExampleUpdateEntry = tuple[Literal["a"], int] | tuple[Literal["b"], str] + ExampleUpdate = ExampleUpdateDict | Iterable[ExampleUpdateEntry] -Security Implications +Type checkers should permit any type compatible with this TypedDict to be passed into the update operation. As with :pep:`589`, they may choose to continue permitting TypedDict types that allow other keys as well, to avoid generating false positives. + +Backwards compatibility +======================= + +This PEP changes the rules for how TypedDict behaves, so code that inspects TypedDict types will have to change to support types using the new features. This is expected to mainly affect type-checkers. + +Security implications ===================== There are no known security consequences arising from this PEP. - How to Teach This ================= -Class documentation should be added to the :mod:`typing` module's documentation, using -that for :class:`~collections.abc.Mapping`, :class:`~typing.Protocol` and -:class:`~typing.TypedDict` as examples. Suggested introductory sentence: "Base class -for read-only mapping protocol classes." +Suggestion for changes to the :mod:`typing` module, in line with current practice: + +* Add this PEP to the others listed. +* Add ``typing.ReadOnly``, linked to TypedDict and this PEP. +* Add the following text to the TypedDict entry: -This PEP could be added to the others listed in the :mod:`typing` module's documentation. +By default, keys not specified in a TypedDict may still be present. Instances can be restricted to only the named keys with the ``other_keys`` flag. *insert example, perhaps using ``in`` to illustrate the benefit* +Individual keys can be excluded from mutate operations using ReadOnly, allowing them to be read but not changed. This is useful when the exact type of the value is not known yet, and so modifying it would break structural subtypes. *insert example* + +If all keys on a TypedDict should be read-only, the ``readonly`` flag can be used as a shorthand. *insert example* Reference Implementation ======================== -No reference implementation exists yet. - +No complete reference implementation exists yet. pyright 1.1.310 ships with a partial implementation of the ReadOnly qualifier. Rejected Alternatives ===================== -Several variations were considered and discarded: +A TypedMapping protocol type +---------------------------- + +:pep:`705` proposed a ``TypedMapping`` protocol type, behaving much like a read-only TypedDict but without the constraint that the runtime type be a ``dict``. The behavior described in this PEP could then be obtained by inheriting a TypedDict from a TypedMapping. This has been set aside for now as more complex, without a strong use-case motivating the additional complexity. + +A higher-order Readonly type +---------------------------- -* A ``readonly`` parameter to ``TypedDict``, behaving much like ``TypedMapping`` but with the additional constraint that instances must be dictionaries at runtime. This was discarded as less flexible due to the extra constraint; additionally, the new type nicely mirrors the existing ``Mapping``/``Dict`` types. -* Inheriting from a ``TypedMapping`` subclass and ``TypedDict`` resulting in mutator methods being added for all fields, not just those actively (re)declared in the class body. Discarded as less flexible, and not matching how inheritance works in other cases for ``TypedDict`` (e.g. total=False and total=True do not affect fields not specified in the class body). -* A generic type that removes mutator methods from its parameter, e.g. ``Readonly[MovieRecord]``. This would naturally want to be defined for a wider set of types than just ``TypedDict`` subclasses, and also raises questions about whether and how it applies to nested types. We decided to keep the scope of this PEP narrower. -* Declaring methods directly on a ``TypedMapping`` class. Methods are a kind of property, but declarations on a ``TypedMapping`` class are defining keys, so mixing the two is potentially confusing. Banning methods also makes it very easy to decide whether a ``TypedDict`` subclass can mix in a protocol or not (yes if it's just ``TypedMapping`` superclasses, no if there's a ``Protocol``). +A generalized higher-order type could be added that removes mutator methods from its parameter, e.g. ``ReadOnly[MovieRecord]``. For a TypedDict, this would be like adding ``readonly=True`` to the declaration. This would naturally want to be defined for a wider set of types than just TypedDict subclasses, and also raises questions about whether and how it applies to nested types. We decided to keep the scope of this PEP narrower. +Preventing other keys with the typing.final decorator +----------------------------------------------------- + +Instead of adding an ``other_keys`` flag to TypedDict, treat classes decorated with :func:`~typing.final` as disallowing other keys. This makes intuitive sense for TypedDict as it stands now: preventing adding any other keys guarantees no other types will be structurally compatible, so it is effectively final. There is also partial support for this idiom in mypy and pyright, which both use it as a way to achieve type discrimination. However, if any keys are read-only, preventing adding any other keys does **not** make the type final any more, so using the decorator this way seems incorrect. We recommend type checkers treat decorating a TypedDict type with final as identical to setting ``other_keys=False``, if they continue to support the idiom for backwards compatibility, but reject any use of final on a TypedDict with read-only keys. Once ``other_keys`` is adopted, they may also wish to deprecate use of final on TypedDicts entirely. + +Using different casing for ``readonly`` keyword or ``ReadOnly`` type +-------------------------------------------------------------------- + +It appears to be common convention to put an initial caps onto words separated by a dash when converting to CamelCase, but to drop the dash completely when converting to snake_case. Django uses ``readonly``, for instance. This appears consistent with the definition of both on Wikipedia: snake_case replaces spaces with dashes, while CamelCase uppercases the first letter of each word. That said, more examples or counterexamples, ideally from the core Python libraries, or better explicit guidance on the convention, would be greatly appreciated. + +Mandate unsound type narrowing +------------------------------ + +The main use-case we are aware of for ``other_keys=False`` (and the current workaround of final-decorated TypedDict types) is to simplify type discrimination, as shown in the motivation section. + +By comparison, TypeScript handles this edge-case by ignoring the possibility of instances of one type in the union having undeclared keys. If a variable is known to be of type ``A | B`` and an ``in`` check is done using a key not explicitly declared on ``B``, it is assumed no instance of ``B`` will pass that check. While technically unsound, this a common enough idiom that it could fall under the recommendation in :pep:`589` that "potentially unsafe operations may be accepted if the alternative is to generate false positive errors for idiomatic code". + +This user request has been rejected multiple times by type checkers, however, suggesting the community prefers strict type-safety over idiomatic code here. Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. + From a1b655a0c7a83af205821c31db955ba749516a57 Mon Sep 17 00:00:00 2001 From: Alice Date: Tue, 19 Sep 2023 08:13:51 +0100 Subject: [PATCH 02/13] Apply Jelle's suggestions from code review Co-authored-by: Jelle Zijlstra --- peps/pep-0705.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/peps/pep-0705.rst b/peps/pep-0705.rst index a763ad2c232..7e4adcebf2a 100644 --- a/peps/pep-0705.rst +++ b/peps/pep-0705.rst @@ -228,7 +228,7 @@ This flag defaults to ``True``. Type-checkers may rely on this restriction:: - def album_keys(album: Album) -> Collection[Literal['name'] | Literal['year']]: + def album_keys(album: Album) -> Collection[Literal['name', 'year']]: # Type checkers may permit this, but should error if Album did not specify `other_keys=False` return album.keys() @@ -245,7 +245,7 @@ Type-checkers should prevent operations that would violate this restriction:: ``readonly`` flag ----------------- -The ``readonly`` flag to ``TypedDict``, when ``True``, indicates that no mutator methods (``__setitem__``, ``__delitem``, ``update``, etc.) will be generated:: +The ``readonly`` flag to ``TypedDict``, when ``True``, indicates that no mutator operations (``__setitem__``, ``__delitem__``, ``update``, etc.) will be permitted:: class NamedDict(TypedDict, readonly=True): name: str @@ -259,7 +259,7 @@ The ``readonly`` flag to ``TypedDict``, when ``True``, indicates that no mutator ``typing.ReadOnly`` flag ------------------------ -The ``typing.ReadOnly`` type qualifier is used to indicate that a variable declared in a ``TypedDict`` definition will not be included in any generated mutator method overloads:: +The ``typing.ReadOnly`` type qualifier is used to indicate that a variable declared in a ``TypedDict`` definition may not be mutated by any operation performed on instances of the ``TypedDict``:: from typing import ReadOnly @@ -370,7 +370,7 @@ Discussion: * All non-specified keys in a type that allows other keys are implicitly of type ``ReadOnly[NotRequired[Any]]``. -* Read-only value types behave covariantly, as they cannot be mutated. This is similar to container types such as ``Mapping``, and different from non-read-only value types, which behave invariantly. Example:: +* Read-only value types behave covariantly, as they cannot be mutated. This is similar to container types such as ``Sequence``, and different from non-read-only value types, which behave invariantly. Example:: class A(TypedDict, readonly=True): x: int | None @@ -384,7 +384,7 @@ Discussion: b: B = {'x': 1} f(b) # Accepted by type checker -* An TypedDict type ``A`` with no explicit key ``'x'`` that allows other keys is not consistent with a TypedDict type with a non-required key ``'x'``, since at runtime the key ``'x'`` could be present and have an incompatible type (which may not be visible through ``A`` due to structural subtyping). The only exception to this rule is if ``'x'`` is non-required, read-only and of type ``Any``. +* A TypedDict type ``A`` with no explicit key ``'x'`` that allows other keys is not consistent with a TypedDict type with a non-required key ``'x'``, since at runtime the key ``'x'`` could be present and have an incompatible type (which may not be visible through ``A`` due to structural subtyping). The only exception to this rule is if ``'x'`` is non-required, read-only and of type ``Any``. * A TypedDict type ``A`` with no key ``'x'`` that does not allow other keys may be consistent with a TypedDict type with a read-only, non-required key ``'x'``. Example:: @@ -494,7 +494,7 @@ Type checkers should permit any type compatible with this TypedDict to be passed Backwards compatibility ======================= -This PEP changes the rules for how TypedDict behaves, so code that inspects TypedDict types will have to change to support types using the new features. This is expected to mainly affect type-checkers. +This PEP adds new features to ``TypedDict``, so code that inspects ``TypedDict`` types will have to change to support types using the new features. This is expected to mainly affect type-checkers. Security implications ===================== @@ -527,7 +527,7 @@ Rejected Alternatives A TypedMapping protocol type ---------------------------- -:pep:`705` proposed a ``TypedMapping`` protocol type, behaving much like a read-only TypedDict but without the constraint that the runtime type be a ``dict``. The behavior described in this PEP could then be obtained by inheriting a TypedDict from a TypedMapping. This has been set aside for now as more complex, without a strong use-case motivating the additional complexity. +An earlier version of :pep:`705` proposed a ``TypedMapping`` protocol type, behaving much like a read-only TypedDict but without the constraint that the runtime type be a ``dict``. The behavior described in the current version of this PEP could then be obtained by inheriting a TypedDict from a TypedMapping. This has been set aside for now as more complex, without a strong use-case motivating the additional complexity. A higher-order Readonly type ---------------------------- From c8c1a129179f1a7c710e9510dc2ff8d117964e37 Mon Sep 17 00:00:00 2001 From: Alice Purcell Date: Tue, 19 Sep 2023 08:17:41 +0100 Subject: [PATCH 03/13] Any keys could also be object or Unknown --- peps/pep-0705.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0705.rst b/peps/pep-0705.rst index 7e4adcebf2a..be0a29c6710 100644 --- a/peps/pep-0705.rst +++ b/peps/pep-0705.rst @@ -384,7 +384,7 @@ Discussion: b: B = {'x': 1} f(b) # Accepted by type checker -* A TypedDict type ``A`` with no explicit key ``'x'`` that allows other keys is not consistent with a TypedDict type with a non-required key ``'x'``, since at runtime the key ``'x'`` could be present and have an incompatible type (which may not be visible through ``A`` due to structural subtyping). The only exception to this rule is if ``'x'`` is non-required, read-only and of type ``Any``. +* A TypedDict type ``A`` with no explicit key ``'x'`` that allows other keys is not consistent with a TypedDict type with a non-required key ``'x'``, since at runtime the key ``'x'`` could be present and have an incompatible type (which may not be visible through ``A`` due to structural subtyping). The only exception to this rule is if ``'x'`` is non-required, read-only and of type ``object`` (or ``Any`` or pylance's ``Unknown``). * A TypedDict type ``A`` with no key ``'x'`` that does not allow other keys may be consistent with a TypedDict type with a read-only, non-required key ``'x'``. Example:: From 313b61ef13b84d7fa23f43fd7ffb63becf8da17f Mon Sep 17 00:00:00 2001 From: Alice Purcell Date: Tue, 19 Sep 2023 08:41:25 +0100 Subject: [PATCH 04/13] Specify inheritance rules for other_keys --- peps/pep-0705.rst | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/peps/pep-0705.rst b/peps/pep-0705.rst index be0a29c6710..1a7598070f3 100644 --- a/peps/pep-0705.rst +++ b/peps/pep-0705.rst @@ -308,6 +308,12 @@ To avoid potential confusion, it is an error to have a read-only type extend a n class BandAlbumAndLabel(BandAndAlbum, readonly=True): # Runtime error label: str + +It is also an error to have a type without ``other_keys`` specified, or with it specified as ``True``, extend a type with ``other_keys=False``:: + + class Person(NamedDict): # Runtime error + age: float + It is valid to have a non-read-only type extend a read-only one. The subclass will not be read-only, but any keys not redeclared in the subclass will remain read-only:: class Album(NamedDict, TypedDict): @@ -348,6 +354,11 @@ Subclasses can also require keys that are read-only but not required in the supe Note that these are just consequences of structural typing, but they are highlighted here as the behavior now differs from the rules specified in :pep:`589`. +Finally, subclasses can have ``other_keys=False`` even if the superclass does not:: + + class Person(OptionalName, other_keys=False): + name: Required[str] + Type consistency ---------------- From 947d0650a27e0f26560e3d3ec25d302338c4d8df Mon Sep 17 00:00:00 2001 From: Alice Purcell Date: Tue, 19 Sep 2023 15:28:27 +0100 Subject: [PATCH 05/13] Address Jukka's feedback --- peps/pep-0705.rst | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/peps/pep-0705.rst b/peps/pep-0705.rst index 1a7598070f3..d5ce183c62e 100644 --- a/peps/pep-0705.rst +++ b/peps/pep-0705.rst @@ -181,7 +181,7 @@ To support this, we propose adding a new boolean flag to ``TypedDict``, ``readon name: str author: str -In addition to these benefits, by flagging arguments of a function as read-only, it makes explicit not just to typecheckers but also to users that the function is not going to modify its inputs, which is usually a desireable property of a function interface. +In addition to these benefits, by flagging arguments of a function as read-only (by using a read-only ``TypedDict`` like ``Movie`` or ``Book``), it makes explicit not just to typecheckers but also to users that the function is not going to modify its inputs, which is usually a desireable property of a function interface. A new ``typing.ReadOnly`` type qualifier allows removing the ability to mutate individual entries, permitting a mixture of readonly and mutable entries. This is necessary for supporting the second motivating example, updating nested dicts:: @@ -548,7 +548,22 @@ A generalized higher-order type could be added that removes mutator methods from Preventing other keys with the typing.final decorator ----------------------------------------------------- -Instead of adding an ``other_keys`` flag to TypedDict, treat classes decorated with :func:`~typing.final` as disallowing other keys. This makes intuitive sense for TypedDict as it stands now: preventing adding any other keys guarantees no other types will be structurally compatible, so it is effectively final. There is also partial support for this idiom in mypy and pyright, which both use it as a way to achieve type discrimination. However, if any keys are read-only, preventing adding any other keys does **not** make the type final any more, so using the decorator this way seems incorrect. We recommend type checkers treat decorating a TypedDict type with final as identical to setting ``other_keys=False``, if they continue to support the idiom for backwards compatibility, but reject any use of final on a TypedDict with read-only keys. Once ``other_keys`` is adopted, they may also wish to deprecate use of final on TypedDicts entirely. +Instead of adding an ``other_keys`` flag to TypedDict, treat classes decorated with :func:`~typing.final` as disallowing other keys. This makes intuitive sense for TypedDict as it stands now: preventing adding any other keys guarantees no other types will be structurally compatible, so it is effectively final. There is also partial support for this idiom in mypy and pyright, which both use it as a way to achieve type discrimination. However, if any keys are read-only, preventing adding any other keys does **not** make the type final any more, so using the decorator this way seems incorrect. For example:: + + class Foo: ... + class Bar(Foo): ... + + @final + class FooHolder(TypedDict, readonly=True): + item: Foo + + @final + class BarHolder(FooHolder, readonly=True): + item: Bar + +Extending a ``TypedDict`` to refine the types is a reasonable feature, but the above code looks like it should raise a runtime error. Should ``@final`` be modified to allow inheritance? Should users be prevented from using this pattern? + +We recommend type checkers treat decorating a TypedDict type with final as identical to setting ``other_keys=False``, if they continue to support the idiom for backwards compatibility, but reject any use of final on a TypedDict with read-only keys. Once ``other_keys`` is adopted, they may also wish to deprecate use of final on TypedDicts entirely. Using different casing for ``readonly`` keyword or ``ReadOnly`` type -------------------------------------------------------------------- From c659f46b38e01f72c0b8b2a8a0da299c17cde6d9 Mon Sep 17 00:00:00 2001 From: Alice Purcell Date: Thu, 21 Sep 2023 17:55:30 +0100 Subject: [PATCH 06/13] Clarify where ReadOnly can be used --- peps/pep-0705.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/peps/pep-0705.rst b/peps/pep-0705.rst index d5ce183c62e..67f34a8df8b 100644 --- a/peps/pep-0705.rst +++ b/peps/pep-0705.rst @@ -189,6 +189,8 @@ A new ``typing.ReadOnly`` type qualifier allows removing the ability to mutate i name: str metadata: ReadOnly[Logs] +This PEP only proposes making ``ReadOnly`` valid in a ``TypedDict``. A possible future extension would be to support it in additional contexts, such as in protocols. + Finally, to support type discrimination, we add a second boolean flag to ``TypedDict``, ``other_keys``, which when set to True, prevents instances from holding any key not explicitly listed in the type:: class EntertainmentMovie(TypedDict, readonly=True, other_keys=False): From f30f4d6f81d3d59e27d57918ba939823277a2fe4 Mon Sep 17 00:00:00 2001 From: Alice Purcell Date: Fri, 22 Sep 2023 11:43:49 +0100 Subject: [PATCH 07/13] Fix NotRequired typos --- peps/pep-0705.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0705.rst b/peps/pep-0705.rst index 67f34a8df8b..bee0a715369 100644 --- a/peps/pep-0705.rst +++ b/peps/pep-0705.rst @@ -372,7 +372,7 @@ A TypedDict type with ``other_keys=False`` is consistent with ``Mapping[str, V]` A TypedDict type ``A`` is consistent with TypedDict ``B`` if ``A`` is structurally compatible with ``B``. This is true if and only if all of the following are satisfied: -* For each key in ``B``, ``A`` has the corresponding key and the corresponding value type in ``A`` is consistent with the value type in ``B``, unless the key in ``B`` is of type ``ReadOnly[NonRequired[Any]]``, in which case it may be missing in ``A`` provided ``A`` allows other keys. +* For each key in ``B``, ``A`` has the corresponding key and the corresponding value type in ``A`` is consistent with the value type in ``B``, unless the key in ``B`` is of type ``ReadOnly[NotRequired[Any]]``, in which case it may be missing in ``A`` provided ``A`` allows other keys. * For each non-read-only key in ``B``, the corresponding value type in ``B`` is also consistent with the corresponding value type in ``A``. * For each required key in ``B``, the corresponding key is required in ``A``. * For each non-read-only, non-required key in ``B``, the corresponding key is not required in ``A``. @@ -491,7 +491,7 @@ For instance:: class Example(TypedDict): a: int - b: NonRequired[str] + b: NotRequired[str] c: ReadOnly[int] class ExampleUpdateDict(TypedDict, total=False, readonly=True, other_keys=False): From 8a799cbfe00f6d053eb8f97bdec9ef213edf475a Mon Sep 17 00:00:00 2001 From: Alice Purcell Date: Fri, 22 Sep 2023 12:04:01 +0100 Subject: [PATCH 08/13] Link to pyright issue 5254 for context --- peps/pep-0705.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/peps/pep-0705.rst b/peps/pep-0705.rst index bee0a715369..5fcdfdbf2d3 100644 --- a/peps/pep-0705.rst +++ b/peps/pep-0705.rst @@ -565,6 +565,8 @@ Instead of adding an ``other_keys`` flag to TypedDict, treat classes decorated w Extending a ``TypedDict`` to refine the types is a reasonable feature, but the above code looks like it should raise a runtime error. Should ``@final`` be modified to allow inheritance? Should users be prevented from using this pattern? +More context for this can be found on `pyright issue 5254 `_. + We recommend type checkers treat decorating a TypedDict type with final as identical to setting ``other_keys=False``, if they continue to support the idiom for backwards compatibility, but reject any use of final on a TypedDict with read-only keys. Once ``other_keys`` is adopted, they may also wish to deprecate use of final on TypedDicts entirely. Using different casing for ``readonly`` keyword or ``ReadOnly`` type From 6c859dcc6b816c52bf08d89ded87c5c1cca61a94 Mon Sep 17 00:00:00 2001 From: Alice Purcell Date: Sat, 23 Sep 2023 17:57:56 +0100 Subject: [PATCH 09/13] Change other_keys=False to other_keys=Never --- peps/pep-0705.rst | 76 ++++++++++++++++++++++++++++++++--------------- 1 file changed, 52 insertions(+), 24 deletions(-) diff --git a/peps/pep-0705.rst b/peps/pep-0705.rst index 5fcdfdbf2d3..c6752878f38 100644 --- a/peps/pep-0705.rst +++ b/peps/pep-0705.rst @@ -191,12 +191,12 @@ A new ``typing.ReadOnly`` type qualifier allows removing the ability to mutate i This PEP only proposes making ``ReadOnly`` valid in a ``TypedDict``. A possible future extension would be to support it in additional contexts, such as in protocols. -Finally, to support type discrimination, we add a second boolean flag to ``TypedDict``, ``other_keys``, which when set to True, prevents instances from holding any key not explicitly listed in the type:: +Finally, to support type discrimination, we add a second flag to ``TypedDict``, ``other_keys``, which when set to ``typing.Never``, prevents instances from holding any key not explicitly listed in the type:: - class EntertainmentMovie(TypedDict, readonly=True, other_keys=False): + class EntertainmentMovie(TypedDict, readonly=True, other_keys=Never): movie: Movie - class EntertainmentBook(TypedDict, readonly=True, other_keys=False): + class EntertainmentBook(TypedDict, readonly=True, other_keys=Never): book: Book Entertainment = EntertainmentMovie | EntertainmentBook @@ -209,34 +209,34 @@ Finally, to support type discrimination, we add a second boolean flag to ``Typed else: raise ValueError("Unexpected entertainment type") +Note this is a subset of the functionality of the `unmerged proposal of PEP-728 `_. + Specification ============= -``TypedDict`` will gain two new boolean flags: ``other_keys`` and ``readonly``. A new ``typing.ReadOnly`` type qualifier is added. +``TypedDict`` will gain two new flags: ``other_keys`` and ``readonly``. A new ``typing.ReadOnly`` type qualifier is added. ``other_keys`` flag ------------------- -The ``other_keys`` flag to ``TypedDict``, when ``False``, indicates that instances may only contain keys explicitly listed in the type:: +The optional ``other_keys`` flag to ``TypedDict`` can have the value ``typing.Never``, indicating that instances may only contain keys explicitly listed in the type:: - class Album(TypedDict, other_keys=False): + class Album(TypedDict, other_keys=Never): name: str year: int class AlbumExtra(Album, TypedDict): band: str # Runtime error -This flag defaults to ``True``. - Type-checkers may rely on this restriction:: def album_keys(album: Album) -> Collection[Literal['name', 'year']]: - # Type checkers may permit this, but should error if Album did not specify `other_keys=False` + # Type checkers may permit this, but should error if Album did not specify `other_keys=Never` return album.keys() Type-checkers should prevent operations that would violate this restriction:: - class AlbumExtra(TypedDict, other_keys=False): + class AlbumExtra(TypedDict, other_keys=Never): name: str year: int band: str @@ -244,10 +244,12 @@ Type-checkers should prevent operations that would violate this restriction:: album: AlbumExtra = { "name": "Flood", year: 1990, band: "They Might Be Giants" } album_keys(album) # Type check error: extra key 'band' +This PEP does not propose supporting any other values than ``other_keys=Never``. Future or concurrent PEPs may extend this flag to permit other types. + ``readonly`` flag ----------------- -The ``readonly`` flag to ``TypedDict``, when ``True``, indicates that no mutator operations (``__setitem__``, ``__delitem__``, ``update``, etc.) will be permitted:: +The optional boolean ``readonly`` flag to ``TypedDict``, when ``True``, indicates that no mutator operations (``__setitem__``, ``__delitem__``, ``update``, etc.) will be permitted:: class NamedDict(TypedDict, readonly=True): name: str @@ -258,6 +260,8 @@ The ``readonly`` flag to ``TypedDict``, when ``True``, indicates that no mutator def set_name(d: NamedDict, name: str) -> None: d["name"] = name # Type check error: cannot modify a read-only entry +The ``readonly`` flag defaults to ``False``. + ``typing.ReadOnly`` flag ------------------------ @@ -280,7 +284,7 @@ Alternative functional syntax The :pep:`alternative functional syntax <589#alternative-syntax>` for TypedDict also supports these features:: - EntityBand = TypedDict('EntityBand', {'band': Band}, readonly=True, other_keys=False) + EntityBand = TypedDict('EntityBand', {'band': Band}, readonly=True, other_keys=Never) BandAndAlbum = TypedDict(`BandAndAlbum', {'band': str, 'album': ReadOnly[Album]}) Interaction with other special types @@ -311,7 +315,7 @@ To avoid potential confusion, it is an error to have a read-only type extend a n label: str -It is also an error to have a type without ``other_keys`` specified, or with it specified as ``True``, extend a type with ``other_keys=False``:: +It is also an error to have a type without ``other_keys`` specified extend a type with ``other_keys=Never``:: class Person(NamedDict): # Runtime error age: float @@ -356,17 +360,17 @@ Subclasses can also require keys that are read-only but not required in the supe Note that these are just consequences of structural typing, but they are highlighted here as the behavior now differs from the rules specified in :pep:`589`. -Finally, subclasses can have ``other_keys=False`` even if the superclass does not:: +Finally, subclasses can have ``other_keys=Never`` even if the superclass does not:: - class Person(OptionalName, other_keys=False): + class Person(OptionalName, other_keys=Never): name: Required[str] Type consistency ---------------- -A TypedDict type with ``other_keys=False`` is consistent with ``Mapping[str, V]``, where ``V`` is the union of all its value types. For instance, the following type is consistent with ``Mapping[str, int | str]``:: +A TypedDict type with ``other_keys=Never`` is consistent with ``Mapping[str, V]``, where ``V`` is the union of all its value types. For instance, the following type is consistent with ``Mapping[str, int | str]``:: - class Person(TypedDict, other_keys=False): + class Person(TypedDict, other_keys=Never): name: str age: int @@ -404,7 +408,7 @@ Discussion: class A(TypedDict, total=False, readonly=True): y: int - class B(TypedDict, other_keys=False): + class B(TypedDict, other_keys=Never): x: int def f(a: A) -> int: @@ -418,10 +422,10 @@ Union Operation The union operation creates a new dictionary with the merged keys and values of its two operands. As such, the result should be consistent with any type that can hold the possible key-value pairs, not just types compatible with the operand types. For example:: - class A(TypedDict, readonly=True, other_keys=False): + class A(TypedDict, readonly=True, other_keys=Never): x: int - class B(TypedDict, total=False, readonly=True, other_keys=False): + class B(TypedDict, total=False, readonly=True, other_keys=Never): x: str class C(TypedDict): @@ -452,7 +456,7 @@ Update Operations Previously, ``clear()`` and ``popitem()`` were rejected by type checkers on TypedDict objects, as they could remove required keys, some of which may not be directly visible because of structural subtyping. However, these methods should be allowed on TypedDicts objects with all keys non-read-only and non-required and with no other keys allowed:: - class A(TypedDict, total=False, other_keys=False): + class A(TypedDict, total=False, other_keys=Never): x: int y: str @@ -494,7 +498,7 @@ For instance:: b: NotRequired[str] c: ReadOnly[int] - class ExampleUpdateDict(TypedDict, total=False, readonly=True, other_keys=False): + class ExampleUpdateDict(TypedDict, total=False, readonly=True, other_keys=Never): a: int b: str # c is not present as it is read-only in Example @@ -567,7 +571,7 @@ Extending a ``TypedDict`` to refine the types is a reasonable feature, but the a More context for this can be found on `pyright issue 5254 `_. -We recommend type checkers treat decorating a TypedDict type with final as identical to setting ``other_keys=False``, if they continue to support the idiom for backwards compatibility, but reject any use of final on a TypedDict with read-only keys. Once ``other_keys`` is adopted, they may also wish to deprecate use of final on TypedDicts entirely. +We recommend type checkers treat decorating a TypedDict type with final as identical to setting ``other_keys=Never``, if they continue to support the idiom for backwards compatibility, but reject any use of final on a TypedDict with read-only keys. Once ``other_keys`` is adopted, they may also wish to deprecate use of final on TypedDicts entirely. Using different casing for ``readonly`` keyword or ``ReadOnly`` type -------------------------------------------------------------------- @@ -577,12 +581,36 @@ It appears to be common convention to put an initial caps onto words separated b Mandate unsound type narrowing ------------------------------ -The main use-case we are aware of for ``other_keys=False`` (and the current workaround of final-decorated TypedDict types) is to simplify type discrimination, as shown in the motivation section. +The main use-case we are aware of for ``other_keys=Never`` (and the current workaround of final-decorated TypedDict types) is to simplify type discrimination, as shown in the motivation section. By comparison, TypeScript handles this edge-case by ignoring the possibility of instances of one type in the union having undeclared keys. If a variable is known to be of type ``A | B`` and an ``in`` check is done using a key not explicitly declared on ``B``, it is assumed no instance of ``B`` will pass that check. While technically unsound, this a common enough idiom that it could fall under the recommendation in :pep:`589` that "potentially unsafe operations may be accepted if the alternative is to generate false positive errors for idiomatic code". This user request has been rejected multiple times by type checkers, however, suggesting the community prefers strict type-safety over idiomatic code here. +Make the ``other_keys`` flag a boolean +-------------------------------------- + +Since ``other_keys`` can only effectively take two values, ``Never`` or absent, it was originally proposed as a boolean flag, with ``other_keys=False`` equivalent to the current ``other_keys=Never``. However, the `unmerged proposal of PEP-728 `_ provides equivalent functionality when restricting other types to ``Never``, so this proposal was updated to use comparable syntax, to make it clearer how the proposals intersect. + +Use a reserved ``__extra__`` key +-------------------------------- + +The `unmerged proposal of PEP-728 `_ proposes different syntax for disallowing other keys:: + + class EntertainmentMovie(TypedDict, readonly=True): + movie: Movie + __extra__: Never + +This new key does not function like other keys -- for instance, it is implicitly ``NotRequired`` but cannot be explicitly marked as such. The author of this PEP prefers the asymmetry of using a keyword argument to set expectations that it does not behave like other key declarations, and others have provided similar feedback on the PR. + +However, this PEP will be updated to match whatever syntax the PEP-728 author decides to go with. + +Leave other_keys to PEP-728 +--------------------------- + +This PEP could drop the ``other_keys`` proposal entirely rather than propose a limited subset of it. However, as this PEP affects the unofficial status-quo of using final to disallow other keys, it seems important to both highlight that issue and propose a solution. + + Copyright ========= This document is placed in the public domain or under the From 0b83d195537929b1ac05f786f830a33e1447b1a7 Mon Sep 17 00:00:00 2001 From: Alice Purcell Date: Sat, 23 Sep 2023 19:06:49 +0100 Subject: [PATCH 10/13] Repeat NamedDict definition --- peps/pep-0705.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/peps/pep-0705.rst b/peps/pep-0705.rst index c6752878f38..62fb6e401b5 100644 --- a/peps/pep-0705.rst +++ b/peps/pep-0705.rst @@ -317,6 +317,9 @@ To avoid potential confusion, it is an error to have a read-only type extend a n It is also an error to have a type without ``other_keys`` specified extend a type with ``other_keys=Never``:: + class NamedDict(TypedDict, readonly=True): + name: str + class Person(NamedDict): # Runtime error age: float From 1c533b57a0bdace18e67ab2b2dbda8eab73a1170 Mon Sep 17 00:00:00 2001 From: Alice Purcell Date: Sat, 23 Sep 2023 19:25:30 +0100 Subject: [PATCH 11/13] Detail interaction with PEP 692 --- peps/pep-0705.rst | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/peps/pep-0705.rst b/peps/pep-0705.rst index 62fb6e401b5..61f2eeec69a 100644 --- a/peps/pep-0705.rst +++ b/peps/pep-0705.rst @@ -511,6 +511,28 @@ For instance:: Type checkers should permit any type compatible with this TypedDict to be passed into the update operation. As with :pep:`589`, they may choose to continue permitting TypedDict types that allow other keys as well, to avoid generating false positives. +Keyword argument typing +----------------------- + +:pep:`692` introduced ``Unpack`` to annotate ``**kwargs`` with a ``TypedDict``. Marking one or more of the entries of a ``TypedDict`` used in this way as read-only will have no effect on the type signature of the method, since all keyword arguments are read-only by design in Python. However, it *will* prevent the entry from being modified in the body of the function:: + + class Args(TypedDict): + key1: int + key2: str + + class ReadonlyArgs(TypedDict, readonly=True): + key1: int + key2: str + + class Function(Protocol): + def __call__(self, **kwargs: Unpack[Args]) -> None: ... + + def impl(self, **kwargs: Unpack[ReadonlyArgs]) -> None: + kwargs["key1"] = 3 # Type check error: key1 is readonly + + fn: Function = impl # Accepted by type checker: function signatures are identical + + Backwards compatibility ======================= From 554f887e7f0a6539776ba3f47b1ff07120d98b17 Mon Sep 17 00:00:00 2001 From: Alice Purcell Date: Sat, 23 Sep 2023 19:31:33 +0100 Subject: [PATCH 12/13] Add cover note to consistency section Make it clear that the new consistency rules should match the existing rules for any types not using the new features. --- peps/pep-0705.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/peps/pep-0705.rst b/peps/pep-0705.rst index 61f2eeec69a..4200d0f0a7a 100644 --- a/peps/pep-0705.rst +++ b/peps/pep-0705.rst @@ -371,6 +371,8 @@ Finally, subclasses can have ``other_keys=Never`` even if the superclass does no Type consistency ---------------- +*This section updates the type consistency rules introduced in* :pep:`589` *to cover the new features in this PEP. In particular, any pair of types that do not use the new features will be consistent under these new rules if (and only if) they were already consistent.* + A TypedDict type with ``other_keys=Never`` is consistent with ``Mapping[str, V]``, where ``V`` is the union of all its value types. For instance, the following type is consistent with ``Mapping[str, int | str]``:: class Person(TypedDict, other_keys=Never): From 202e0d5a825376671fe7ed1466b3d62a88a4101d Mon Sep 17 00:00:00 2001 From: Alice Purcell Date: Sat, 23 Sep 2023 19:39:23 +0100 Subject: [PATCH 13/13] Nod to pyright's use of Unknown over Any --- peps/pep-0705.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0705.rst b/peps/pep-0705.rst index 4200d0f0a7a..15bcfe8afc0 100644 --- a/peps/pep-0705.rst +++ b/peps/pep-0705.rst @@ -390,7 +390,7 @@ A TypedDict type ``A`` is consistent with TypedDict ``B`` if ``A`` is structural Discussion: -* All non-specified keys in a type that allows other keys are implicitly of type ``ReadOnly[NotRequired[Any]]``. +* All non-specified keys in a type that allows other keys are implicitly of type ``ReadOnly[NotRequired[Any]]`` (or ``ReadOnly[NotRequired[Unknown]]`` in pyright). * Read-only value types behave covariantly, as they cannot be mutated. This is similar to container types such as ``Sequence``, and different from non-read-only value types, which behave invariantly. Example::