Use union instead of join for unifying types

[JelleZijlstra]

@erictraut has gathered a long list of problems caused by mypy's behavior of using the "join" operator to unify types in various contexts.

• Incompatible return value type (got "function", expected "Callable[..., Any]") #12053
• Attribute of type A | B | None becomes object after narrowing checks #12009
• enumerate causes incompatible type mypy error #11934
• a joining of two types becomes object not a union #11440
• Unable to remove Optional wrapper when using Mapping[str, str] instead of dict #11618
• Expression has type "function", variable has type "Callable[[], None]" #10740
• Join of types depends on order of types #10442
• Incorrect type reification on optional union attribute in conditional #7884
• mypy infers conditional expression of Dict[str, object] to be object instead of union #7835
• Use Union for tuple fallback? #7616
• Wrong type inference (object instead of List) #6968
• Union types for dict keys are not inferred in union context #6079
• Conditional expression in "with" statement inferred as 'object', lacking __enter__ and __exit__ #5512
• False exception error in raise statement with a ternary conditional operator #5128
• Constrained TypeVar fails to narrow on left hand side of conditional expression #4134
• join is not associative in the presence of unions #3339

I believe that we should change this behavior.

If someone opens a PR to do this, it would be interesting to see what the results look like in mypy-primer and mypy's CI.

[A5rocks]

How will this interact with TypeForm vs Type?

Does this mean join(A,B) will still be accepted for Type[T]? Union[A, B] is not.

Nevermind: I was being confused. This doesn't matter I believe. (TypeForm is more about passing a type in, not about the type of an object)

[bluetech]

I noticed that this was tried for the specific case of conditional expressions (type of x if y else z) in 532f3fb (issue #3487) but was partly reverted in 94309a1 because it "caused many problems in internal Dropbox repos that we aren't ready to fix yet.".

[KotlinIsland]

Basedmypy uses union. We have seen great success with it. (we also form intersections at meets)

[JukkaL]

I agree that we should probably do this, but I also want to see the fallout first. Due to our backward compatibility policy, the change needs to be first introduced behind a feature flag, and we need a major mypy release when we switch the default (e.g. 2.0). This will let us iterate on the change and allow users to experiment with it easily before we make the switch.

[ilevkivskyi]

Just to add my 50 cents here: I don't like the idea of completely switching to unions. The fact that this will fix 20 issues doesn't guarantee that we will not get 40 new issues caused by things like:

def basket() -> set[Fruit]:
    res = {Banana(), Apple(), Orange()}
    # do something with res...
    return res

I think a better (and probably safer) approach is: we should infer unions in "heterogeneous" contexts, while keeping joins in "homogeneous" contexts. By "heterogeneous" contexts I mean things like tuple types, ternary expressions, user declared unions, star arguments (which are also tuples under the hood), maybe there are some more. Note that a significant amount of issues mentioned above are about these cases. We can try switching these cases one by one, and see what is the fallout.

My prediction is that fallout from switching tuple fallback to union will be minimal (in particular since tuple instance is covariant), we can start with this.

[KotlinIsland]

@ilevkivskyi In my experience, we haven't seen any issues with this functionality in basedmypy. You even get a very useful message in the example you provided:

class Fruit: ...
class Apple(Fruit): ...
class Orange(Fruit): ...
class Banana(Fruit): ...

def f() -> set[Fruit]:
    result = {Apple(), Orange(), Banana()}
    return result  # test.py:26: error: Incompatible return value type (got "set[Apple | Orange | Banana]", expected "set[Fruit]")  [return-value]
                   # test.py:26: note: Perhaps you need a type annotation for "a"? Suggestion: "set[Fruit]"

Just some opinions from the basedmypy team is that TypeScript always performs a unionization here as well.

[asottile-sentry]

because I'm curious -- is there any trick to adjust mypy's inference to resolve to the union that's shorter than enumerating all the types?

class Base: pass
class A(Base): pass
class B(Base): pass
class C(Base): pass

x = [A, B, C]  # implicitly `list[type[Base]]`
y: list[type[A | B | C]] = [A, B, C]  # works but is a little cumbersome as the number increases

[ilevkivskyi]

I am actually going to add a flag so that people can opt in if they want more unions (the default however is unlikely to change in next few years).

[mishamsk]

@ilevkivskyi what's the best way to express support for the union behavior? Also, judging by your phrasing, may I assume that mypy may get the "union way" under as an opt-in sooner rather than later?