TypedDict cannot be used where a normal dict is expected

[TimSimpsonR]

This is probably intended behavior, and if so it be good to have the motivation documented somewhere alongside TypedDict's current docs. Apologies if I'm missing something in the docs.

I would expect to be able to pass a TypedDict to any function that expects a plain old dict:

    import mypy_extensions as mt

    FancyDict = mt.TypedDict('FancyDict', {'a': str})

    fancy = FancyDict(a='1')

    def use_any_dict(arg: dict) -> None:
        print(arg)

    use_any_dict(fancy)

However the above snippet fails:

    error: Argument 1 to "use_any_dict" has incompatible type "FancyDict"; expected "Dict[Any, Any]"

TypedDict is a plain dictionary at runtime, which is what the signature of use_any_dict says it accepts. So why doesn't MyPy allow it?

Going the other direction- passing a Dict[Any,Any] to something expecting a certain TypedDict- also fails, which makes sense to me. But it leaves me wondering how code using normal dicts is supposed to interop with newer code using TypedDict aside from using casts or # type: ignore.

I've tested this with mypy 0.590 from PyPi as well as commit d6a22cf13a5a44d7db181ae5e1a3faf2c55c02b4 from trunk. I'm invoking MyPy as follows:

    mypy     \
        --ignore-missing-imports    \
        --strict-optional     \
        --check-untyped-defs     \
        --disallow-untyped-calls     \
        --disallow-incomplete-defs     \
        --disallow-untyped-defs \
        my_module

[JelleZijlstra]

This is probably related to invariance. See http://mypy.readthedocs.io/en/latest/common_issues.html#invariance-vs-covariance.

[JukkaL]

Yeah, this is a bit unfortunate, and it's related to invariance. You can replace dict with Mapping in the type annotation and things should work.

[TimSimpsonR]

Thank you very much. Using Mapping works for me.

I'm ready to make a pull request that adds a mention of this to the docs for TypedDict if that helps. Otherwise I'm ok with closing this.

[jasonkuhrt]

Can someone show a working code sample?

[rgant]

I think it's ridiculous to have to go through this, but this is a working method of handling a library function that only accepts Dict but you want to use TypedDict:

import csv
import typing

from mypy_extensions import TypedDict


User = TypedDict('User', {  # pylint: disable=invalid-name
    'uid': str,
    'email': str,
})

def write_to_csv(users: typing.List[User], outfile: str) -> None:
    """ Writes users list to CSV file. """
    fieldnames = list(users[0].keys())

    with open(outfile, 'w', newline='') as csvfile:
        writer = csv.DictWriter(outfile, fieldnames)
        writer.writeheader()

        for usr in users:
            # Cast to Dict until TypedDict is Dict compatable
            # https://github.com/python/mypy/issues/4976 
            usr_dict = typing.cast(typing.Dict[str, typing.Any], usr)
            writer.writerow(usr_dict)

Without the usr_dict typing.cast() you get the error:

Argument 1 to "writerow" of "DictWriter" has incompatible type "User"; expected "Dict[str, Any]"

[ilevkivskyi]

@JukkaL can we do something here, or should we just close this issue?

[JukkaL]

Making TypedDicts compatible with Dict[str, Any] would be nice, but it's unclear what's the best way to do this. We could special case this in subtype checks, but then the behavior under joins would be inconsistent. Joins would use the fallback type, so would generally produce Mapping types. Mapping is not compatible with Dict[str, Any], which results in the main inconsistency that I can see. Changing the TypedDict fallback type doesn't sound like a good idea to me.

Another idea would be to change typeshed signatures that only accept Dict[str, Any] to Mapping[str, Any] whenever it makes sense. If some functions don't accept arbitrary mappings, we wouldn't be able to do this, however.

Any thoughts?

[ilevkivskyi]

I think it is probably OK to special-case Dict[str, Any] (if one puts Any, not object there, then one already accepts the risk). However, I think we can just wait a bit more to see how popular is this request.

[rgant]

I guess I just don't understand mypy's plans for Dict and TypedDict. Because I mostly deal with JSON APIs, Dict is not something I ever intend to use (excepting casting). I guess I just assumed that eventually Dict and TypedDict would be merged.

I see in the documentation explaining why TypedDict doesn't extend Dict:

A TypedDict object is not a subtype of the regular Dict[...] type (and vice versa), since Dict allows arbitrary keys to be added and removed, unlike TypedDict.

But I don't know why a subtype of Dict cannot disallow arbitrary keys. I also don't see why TypedDict cannot accept arbitrary keys. TypeScript interfaces support all the complicated objects I've ever thrown at it.

After all TypedDict isn't a Python-ism, it's a mypy solution to a lack of Dict's ability to accept complicated patterns. I was really hoping that eventually I would be able to do something like typing.Dict[{'uid': str, 'email': str,}]

If we cannot do that, then can we just have TypedDict figure out the equivalent Dict syntax?

User = TypedDict('User', {  # pylint: disable=invalid-name
    'id': int,
    'email': str,
    'birthdate': datetime.date,
    'meta': UserMeta # A TypedDict
}) # == typing.Dict[str, typing.Union[int, str, datetime.date, UserMeta]]

Or add a third parameter that allows me to manually specify the Dict type of the TypedDict:

User = TypedDict('User', {  # pylint: disable=invalid-name
    'id': int,
    'email': str,
    'birthdate': datetime.date,
    'meta': UserMeta # A TypedDict
}, typing.Dict[str, typing.Any])

I just think it's very confusing to have to create a new variable to cast a variable for the type checker to be happy.

[JukkaL]

But I don't know why a subtype of Dict cannot disallow arbitrary keys.

In your example, if User would be compatible with the Dict type, it looks like you could do things like x['id'] = 'abc' through the Dict type, which would break type safety, since the value of the 'id' key is supposed to be an int. Also, a TypedDict subtype of User should define additional keys with values that are not covered by the union, so accessing .values() or .items() through the dict type would be unsafe as well.

I'm not quite sure what you mean by equivalent Dict syntax, as the two types don't seem to be equivalent. Can you provide a more complete code example that shows what isn't working right now as you'd expect, and explain how your proposal would resolve the issue? And maybe you can also explain how you'd deal with the issue in TypeScript.

[rgant]

In typescript I can do a mixture of known and unknown properties on my object like this:

interface MyInterface {
  fixedKey1: string,
  fixedKey2: number,
  [x: string]: string | number, 
}

And in my head I would translate that to something like:

MyInterface = TypedDict('MyInterface', 
    {'fixedKey1': str, 'fixedKey2': int},
    arbitrary=[str, Union[str, int]]
)

Then I could use the type like:

payload: MyInterface  = {'fixedKey1': 'foo', 'fixedKey2': 123, 'meta': 'bar'}

The important part being that I cannot make fixedKey2 a string as my API doesn't allow for that. But there could be other data that I haven't specified yet, or that my partner wants to shove in that my type checker could then handle.

[gricey432]

In my case, if I replace dict with Mapping all the errors go away on the function calls, however in my function I use .pop and now get a type error for that.

from typing import Mapping
from typing_extensions import TypedDict

def myfunc(d: Mapping):
    d.pop("key")

TD = TypedDict("TD", {"b": int})
b: TD = {"b": 2}
myfunc(b)

error: "Mapping[Any, Any]" has no attribute "pop"

If I replace Mapping with MutableMapping the .pop is now happy but the original problem is back.

error: Argument 1 to "myfunc" has incompatible type "TD"; expected "MutableMapping[Any, Any]"

[JukkaL]

@gricey43 One of the reasons why TypedDict types are compatible with Mapping is that it doesn't provide any mutation operations such as pop. What you want to do is not type safe, so you'll need to work around the type system by using a cast, for example.

[JukkaL]

After more practical experience with TypedDicts, not having TypedDict compatible with Dict[str, Any] can actually be quite useful when trying to ensure that all dictionaries in some code are precisely checked. In any case, the current situation seems acceptable, and I'm not convinced that it's worth adding special cases to type checking for this specific use case.

[gh-andre]

Mapping, suggested above as a workaround, won't work with MutableMapping either. For example, having code working with pymongo, one would use functions taking MutableMapping for Mongo documents, because Mongo client may insert _id field if it doesn't exist.

from typing import MutableMapping, Mapping
from typing import Any
from typing_extensions import TypedDict, NotRequired

class X(TypedDict):
    i: int
    s: str
    f: NotRequired[float]

def mongodb_fn(x: MutableMapping[str, Any]) -> None:
    pass

x: X = {"i" : 1, "s" : "A"}

# no warning here - it's mutable
x["i"] = 23

# Argument 1 to "mongodb_fn" has incompatible type "X"; expected "MutableMapping[str, Any]"
mongodb_fn(x)

TypedDict instances work very well for Mongo documents because of the explicit Required/NotRequired, as opposed to having None in classes or in @dataclasses for missing attributes, which doesn't work as intended.

Also, PEP 589 explicitly says TypedDict is mutable:

Value types behave invariantly, since TypedDict objects are mutable. This is similar to mutable container types such as List and Dict. Example where this is relevant:

https://peps.python.org/pep-0589/#type-consistency

I'm hoping closing this issue can be reconsidered at some point.

[boompig]

I hope maintainers will revisit opening this issue, since the primary usecase of TypedDict is a drop-in replacement for a plain dict in code. If you use TypedDict in that way, then you (a) encounter this somewhat bizarre issue; and (b) have to go through your entire codebase changing all type annotations from dict to Mapping. This to me seems contrary to the spirit of the reason why TypedDict was introduced in the first place.

Consider some large codebase with many functions of the form

def accept_foo(user: User, metadata: dict) -> None:
    ...

Now you want to add some restrictions on metadata so you convert it to a TypedDict. Then everywhere else in the code, you have to convert the type annotations on metadata. This makes adding gradual typing quite onerous, and is quite a bit less friendly than languages like TypeScript where this would naturally work.

[gh-andre]

Now you want to add some restrictions on metadata so you convert it to a TypedDict. Then everywhere else in the code, you have to convert the type annotations on metadata.

Isn't it the whole point of static type checking, so it validates all the places in the code interacting according to the same contract?

quite a bit less friendly than languages like TypeScript where this would naturally work

Not sure how. TypeScript compiler would error out while compiling mismatching types and in Python all static type checkers are 3rd-party tools and the interpreter just skips types. TypeScript would definitely report as a warning a plain object used instead of Record<string, any>.

[erictraut]

@boompig, the typing rules for TypedDict were established in PEP 589 more than four years ago. A change like the one you're suggesting would typically be discussed in the python/typing discussion forum. If agreed upon, it would then be adopted by all Python type checkers, not just mypy.

It's unlikely that the behavior of TypedDict would be changed at this point. Doing so would break code that assumes the current rules as documented in PEP 589. For example, if someone has defined an overloaded function where the first signature accepts a dict and a second signature accepts a TypedDict, the overload matching behavior would change if the type compatibility rules were modified for TypedDict.

@gh-andre, here's an example in TypeScript that demonstrates what @boompig is referring to. This code doesn't generate any errors in TypeScript.

function foo(obj: { [key: string]: any }) {
    obj.a = 'hi';
}

const x: { a: number } = { a: 1 };
foo(x);
console.log(x.a); // Prints 'hi'

TypeScript accommodates unsafe behaviors in the case where any is used. In general, the Python type system likewise accommodates unsafe behaviors when Any is used, so I find it a bit inconsistent that TypedDict isn't compatible with dict[str, Any] (or more generally, with dict[str, <union of value types in TypedDict>]). I don't know the full reasoning behind that decision.

[gh-andre]

the typing rules for TypedDict were established in PEP 589 more than four years ago.

PEP589 says that TypedDict instances are mutable, which makes them MutableMapping with a fixed set of keys, rather than Mapping, as the current implementation enforces.

[heitorlessa]

Just bumped into this where we modify a log record typed with TypedDict and Mypy fails as described in this issue.

from typing import TypedDict

class LogRecord(TypedDict):
    message: str

log_record: LogRecord = {
    "message": "something happened"
}

# simplifying a much longer code use case
log_record["event"] = log_record.pop("message")

Mypy error

error:11 TypedDict "LogRecord" has no key "event"  [typeddict-unknown-key]
error:11 Key "message" of TypedDict "LogRecord" cannot be deleted  [misc]

[gh-andre]

@heitorlessa
Just as a quick note, for the top error, you need NotRequired for event, like this:

from typing_extensions import TypedDict, NotRequired

class LogRecord(TypedDict):
    message: str
    event: NotRequired[str]

log_record: LogRecord = {
    "message": "something happened"
}

log_record["event"] = log_record.pop("message")

[heitorlessa]

Thanks Andre :) I should’ve provided more context — the example is for when customers want to rename that field, e.g., customer wants to bring their own logging standards. We use NotRequired for other fields (it’s great!)

…

On Fri, 9 Jun 2023 at 20:03, Andre ***@***.***> wrote: @heitorlessa <https://github.com/heitorlessa> Just as a quick note, for the top error, you need NotRequired for event, like this: from typing_extensions import TypedDict, NotRequired class LogRecord(TypedDict): message: str event: NotRequired[str] log_record: LogRecord = { "message": "something happened" } log_record["event"] = log_record.pop("message") — Reply to this email directly, view it on GitHub <#4976 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBGABZ674ZCEWP7STR3XKNQPXANCNFSM4E4WJW3Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[gh-andre]

@heitorlessa Now that I'm thinking, you also need NotRequired on message because it's being deleted from the dictionary. This will take care of the second error.

[heitorlessa]

thanks again @gh-andre, that wouldn't be accurate in our case tho, since message is required. Full context and example for the example we're upgrading with TypedDict - in this case, we're purposefully removing a key with pop to showcase a remap of a dict key (log record).

we're gonna add two specific ignores in this case now. Appreciate the care either way

[gh-andre]

@heitorlessa That's one of the challenges of TypedDict in that one class may not be enough for all cases. For example, reading documents from Mongo DB, which are defined via a fairly elaborate JSON schema and some parts of those documents may be missing or not as defined by other values. JSON schema allows for that, but using a single TypedDict won't work for this.

I opted for some code duplication in having multiple TypedDict for different use cases and using wrapping classes for working with typed dictionaries that provide meaningful methods. It's not ideal, but that's the best approach I found so far. Some of that is outlined in the issue below, which would help with some of duplication in having to repeat field names

python/typing#1394

, but I can see use cases in which such duplication would be a non-starter because of increased maintenance.