Remove unnecessary overloads from re

[jpy-git]

This PR follows on from the discussion here where @AlexWaygood highlighted that overloads are unnecessary when a function has the same return type and therefore simple union-types should be used instead. This PR combines these unnecessary overloads in re.pyi and for Pattern in typing.pyi.

Additionally I spotted that Pattern.split should have a return type of list[AnyStr | Any] rather than list[AnyStr]:

(re.split is already correct)

[srittau]

The primer output looks a bit concerning.

primer output

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/strings/object_array.py:216: error: Value of type variable "AnyStr" of "compile" cannot be "Union[str, Pattern[Any], Any]"  [type-var]
+ pandas/core/array_algos/replace.py:88: error: Value of type variable "AnyStr" of "search" cannot be "Union[str, Pattern[Any], Any]"  [type-var]

sphinx (https://github.com/sphinx-doc/sphinx)
+ sphinx/util/docstrings.py:20:22: error: Value of type variable "AnyStr" of "compile" cannot be "Union[str, Pattern[Any], Any]"
+ sphinx/util/docstrings.py: note: In function "separate_metadata":
+ sphinx/util/docstrings.py:39:30: error: Value of type "Union[str, Pattern[Any], Any]" is not indexable
+ sphinx/testing/util.py: note: In function "assert_re_search":
+ sphinx/testing/util.py:34:12: error: Value of type variable "AnyStr" of "search" cannot be "Union[str, Pattern[Any], Any]"
+ sphinx/testing/util.py: note: In function "assert_not_re_search":
+ sphinx/testing/util.py:39:8: error: Value of type variable "AnyStr" of "search" cannot be "Union[str, Pattern[Any], Any]"

[AlexWaygood]

Well, I can't say I really understand why a union-type causes mypy to produce these false-positives when overloads don't. It doesn't make much sense to me.

[hauntsaninja]

AnyStr is a type variable, so mypy might be struggling to solve T | Pattern[T] == str | Pattern[Any]. I'm not convinced this isn't a mypy bug (e.g. unions can cause ambiguity when solving type vars, Any is a tricky type, value restricted type vars are tricky), but mypy's confusion is at least understandable. Would be worth checking how other type checkers deal with this.

[hauntsaninja]

Came back to this after eating a lunch, I'm now fairly sure this is a mypy bug, but I'd still be curious to see if other type checkers do the right thing.

from typing import Any, TypeVar, Union

T = TypeVar("T")
V = TypeVar("V", str, bytes)

def check_t(x: T | list[T]) -> T: ...
def check_v(x: V | list[V]) -> V: ...

x_str: str
x_list_any: list[Any]
x_str_list_any: str | list[Any]

# mypy does the correct thing here

reveal_type(check_t(x_str))  # str
reveal_type(check_t(x_list_any))  # list[Any] | Any
reveal_type(check_t(x_str_list_any))  # str | list[Any] | Any
reveal_type(check_v(x_str))  # str

# but maybe not here

# E: Value of type variable "V" of "check_v" cannot be "Union[List[Any], Any]"
# N: Revealed type is "Union[builtins.list[Any], Any]"
reveal_type(check_v(x_list_any))
# E: Value of type variable "V" of "check_v" cannot be "Union[str, List[Any], Any]"
# N: Revealed type is "Union[builtins.str, builtins.list[Any], Any]"
reveal_type(check_v(x_str_list_any))

Looking at the output, my guess is that mypy isn't taking value restrictions into account when solving constraints and only using value restriction as a check later. That is, list[Any] cannot match the T part of T | list[T] because it would violate the value restriction.

[jpy-git]

What I find a bit weird about these mypy_primer errors is that the changes in this PR are basically the same as what we added for regex in #6713.

If we take this error from the mypy_primer output:
sphinx/util/docstrings.py:20:22: error: Value of type variable "AnyStr" of "compile" cannot be "Union[str, Pattern[Any], Any]
It happens on this line:
https://github.com/sphinx-doc/sphinx/blob/eed0730b4ba3bd2fbd34f2d6ab555ba876c77717/sphinx/util/docstrings.py#L20
but if we repeat that test but using regex instead, which has similar union typing pattern: AnyStr | _regex.Pattern[AnyStr], we don't see any errors as expected:

from docutils.parsers.rst.states import Body
import regex

s = Body.patterns["field_marker"]
reveal_type(s)
p = regex.compile(s)
reveal_type(p)

>>> mypy test.py
test2.py:5: note: Revealed type is "Any"
test2.py:7: note: Revealed type is "regex._regex.Pattern[Any]"

[hauntsaninja]

That's probably related to your setup, e.g. looks like you might be missing docutils-stubs (note, different from types-docutils) or custom typeshed dir. I can repro with your test.py just fine.
The code in my previous comment is also a self contained repro (replace list -> Pattern, V -> AnyStr to make it resemble the original case)

[jpy-git]

Found this comment that seems to be referring to it being a known issue with mypy not being able to resolve T | Cls[T] and hence the overloads:
#5470 (comment)

Wasn't able to track down a corresponding issue in mypy though

[hauntsaninja]

I opened python/mypy#11880 to track this. As I mentioned, the issue seems clearly related to value restriction

[srittau]

I marked this as "deferred" for now, until the mypy problems are sorted out. PR seems good otherwise.

[JelleZijlstra]

I'm not hopeful that the mypy bug will be resolved any time soon. It doesn't seem like this PR fixes any concrete problem, so can we just close it?