Fix type argument inference for overloaded functions with explicit self types (Fixes #14943).

[tyralla]

Fix type argument inference for overloaded functions with explicit self types (Fixes #14943).

When finding no compatible callable right away, try to let method ConstraintBuilderVisitor.find_matching_overload_item return the first overloaded function with the suitable self type.

Checked by test case TestOverloadedMethodWithExplictSelfTypes.

[JukkaL]

The definition attribute is only supposed to be used for error messages, and it's not guaranteed to be set. It cannot be reliably used for type checking. It's not serialized, in particular, but there may well be other contexts where it's lost that I'm not aware of.

Also, the bound_args attribute is unfortunately not documented (or the documentation seems incorrect). Again, I'm not sure if we can expect it to be always set.

Here's an alternative approach that might work. When we bind the self type to an overloaded method (in mypy.typeops.bind_self), what about filtering the overload items at that point to those matching the self type (only when explicit self types are used)? This filtering would happen in an earlier stage, and I believe we have access to all the necessary information without using undocumented or inconsistently set attributes. The result of the filtering could be a non-overloaded callable type or an overloaded type with fewer items.

[tyralla]

Thank you for your feedback. Using two fragile attributes in five new lines - what a negative hit rate...

Your suggestion targets code sections I did not encounter yet. My first naive implementation of your suggestion solved the original issue but broke many other things. So far, I could not find any information to distinguish implicit and explicit self types within bind_self. Is there some obvious way I am missing?

I would take a second, more thorough try within the next few days. Please do not hesitate to fix this yourself if you already have a good solution in mind and if you want that issue fixed before the 1.2 release.

[tyralla]

I tried to filter within bind_self as suggested. All tests pass (on my computer). Let's see what Mypy primer says.

I am still not overly happy with my approach for the following reasons:

1. I added the ignore_type_vars option to is_subtype, which seems like a huge change for such a specific problem to me.
2. I found no way to distinguish explicit self types from implicit self types.
3. The result is still always an overloaded type (changing this within bind_self would be simple but requires changing its signature and eventually many code sections that use bind_self).

[tyralla]

3. The result is still always an overloaded type (changing this within bind_self would be simple but requires changing its signature and eventually many code sections that use bind_self).

I played around with returning a single non-overloaded callable for a few minutes. Maybe it's not a good idea because it changes the signatures in error reports and notes. On the other hand, I did not encounter a case where an overloaded type with a single item caused a problem (but I did not actively search for such a problem).

[hauntsaninja]

(drive by comment, will look further when I get time)
Hmm, I'd added logic like this in #14017 and later moved it to bind_and_map_method in #14882. Maybe there's some unification to be had

[tyralla]

Hmm, I'd added logic like this in #14017 and later moved it to bind_and_map_method in #14882. Maybe there's some unification to be had

Your tip seems helpful. I could remove the filtering logic you implemented in checker.TypeChecker.bind_and_map_method without breaking any tests. Also, testSelfTypeOverrideCompatibilityTypeVar, previously marked by "xfail", now passes. However, this benefit still comes at the cost of the ignore_type_vars approach (which might eventually shadow some actual bugs?).

[hauntsaninja]

Yeah, I'm not sure about the ignore_type_vars

See this other related PR I opened here: #15045 In this case I was able to replace type vars by their upper bound

[tyralla]

I added the test case you introduced in #15045 for testing (I can remove it later). It passes with the current ignore_type_vars approach.

For further testing, I also disabled the ignore_type_vars option (not committed). This results only in problems for the test cases testSelfTypeRestrictedMethodOverloadInit and testSelfTypeRestrictedMethodOverloadInitFallBacks, which check the following modifications of the self type:

class C(Generic[T]):
    @overload
    def __init__(self: C[T], item: T, use_tuple: Literal[False]) -> None: ...
    @overload
    def __init__(self: C[Tuple[T, ...]], item: T, use_tuple: Literal[True]) -> None: ...

Maybe assuming general TypeVar compatibility in the given context is not such a bad thing because Tuple[T, ...]] could be replaced by arbitrarily complex expressions. And actual bugs (e.g. because T is bound to something else) should be found elsewhere. Or can you think of an example where the ignore_type_vars approach would not remove an overloaded item with a well-defined explicit self type that does not fit the current "original type"?

[hauntsaninja]

I think that test case is buggy, there was some discussion at #14729 (comment) and python/typing#1340 (comment)

[tyralla]

Interesting. If we would disallow such self type definitions, the ignore_type_vars option would hopefully become obsolete. I commit this change to see the results on the primer.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/series.py:3330: error: Signature of "sort_values" incompatible with supertype "NDFrame"  [override]
+ pandas/core/series.py:3330: note:      Superclass:
+ pandas/core/series.py:3330: note:          @overload
+ pandas/core/series.py:3330: note:          def sort_values(self, *, axis: Union[int, Literal['index', 'columns', 'rows']] = ..., ascending: Union[bool, Sequence[bool]] = ..., inplace: Literal[False] = ..., kind: str = ..., na_position: str = ..., ignore_index: bool = ..., key: Optional[Callable[[Series], Union[Series, Union[Union[ExtensionArray, ndarray[Any, Any]], Index, Series]]]] = ...) -> Series
+ pandas/core/series.py:3330: note:          @overload
+ pandas/core/series.py:3330: note:          def sort_values(self, *, axis: Union[int, Literal['index', 'columns', 'rows']] = ..., ascending: Union[bool, Sequence[bool]] = ..., inplace: Literal[True], kind: str = ..., na_position: str = ..., ignore_index: bool = ..., key: Optional[Callable[[Series], Union[Series, Union[Union[ExtensionArray, ndarray[Any, Any]], Index, Series]]]] = ...) -> None
+ pandas/core/series.py:3330: note:          @overload
+ pandas/core/series.py:3330: note:          def sort_values(self, *, axis: Union[int, Literal['index', 'columns', 'rows']] = ..., ascending: Union[bool, Sequence[bool]] = ..., inplace: bool = ..., kind: str = ..., na_position: str = ..., ignore_index: bool = ..., key: Optional[Callable[[Series], Union[Series, Union[Union[ExtensionArray, ndarray[Any, Any]], Index, Series]]]] = ...) -> Optional[Series]
+ pandas/core/series.py:3330: note:      Subclass:
+ pandas/core/series.py:3330: note:          @overload
+ pandas/core/series.py:3330: note:          def sort_values(self, *, axis: Union[int, Literal['index', 'columns', 'rows']] = ..., ascending: Union[bool, int, Sequence[bool], Sequence[int]] = ..., inplace: Literal[False] = ..., kind: str = ..., na_position: str = ..., ignore_index: bool = ..., key: Optional[Callable[[Series], Union[Series, Union[Union[ExtensionArray, ndarray[Any, Any]], Index, Series]]]] = ...) -> Series
+ pandas/core/series.py:3330: note:          @overload
+ pandas/core/series.py:3330: note:          def sort_values(self, *, axis: Union[int, Literal['index', 'columns', 'rows']] = ..., ascending: Union[bool, int, Sequence[bool], Sequence[int]] = ..., inplace: Literal[True], kind: str = ..., na_position: str = ..., ignore_index: bool = ..., key: Optional[Callable[[Series], Union[Series, Union[Union[ExtensionArray, ndarray[Any, Any]], Index, Series]]]] = ...) -> None

pandas-stubs (https://github.com/pandas-dev/pandas-stubs)
- pandas-stubs/core/indexes/range.pyi:87: error: Unused "type: ignore" comment

cwltool (https://github.com/common-workflow-language/cwltool)
- cwltool/main.py:964: error: Unused "type: ignore[arg-type]" comment

mongo-python-driver (https://github.com/mongodb/mongo-python-driver)
- bson/son.py: note: At top level:
- bson/son.py:118: error: Unused "type: ignore" comment

[tyralla]

The ignore_type_vars=False experiment showed that type_object_type_from_function also required some adjustment, as it relies on pre-determined self types. Many problems reported by the primer (at least those related to defaultdict) were due to filtering the overloaded items but not their pre-determined self types. I fixed this, but so far likely not in the best possible style.

However, some other findings gave me the impression that the ignore_type_vars approach might really be okay, so I reactivated it. Everything else I tried had its edges but no advantages I am aware of.