typing: accept buffers in typing.IO.write #9084

JelleZijlstra · 2022-11-04T03:02:37Z

JelleZijlstra · 2022-11-04T03:11:45Z

The mypy failures are a bug, python/mypy#14002.

srittau · 2022-11-04T11:31:56Z

This looks quite disruptive to me and I'm not sure whether it's worth it, considering the slightly legacy nature of IO.

JelleZijlstra · 2022-11-04T15:45:15Z

Worth noting that most of the mypy-primer fallout is due to a mypy bug (python/mypy#14002).

hauntsaninja · 2022-11-05T22:33:50Z

This seems useful, but given that I think I have a mypy fix, let's wait until it's released: python/mypy#14017

AlexWaygood

Looks like we can go back to this now, with the latest mypy release!

AlexWaygood · 2023-02-08T08:44:40Z

stdlib/codecs.pyi

@@ -239,7 +239,7 @@ class StreamReaderWriter(TextIO):
    def __next__(self) -> str: ...
    def __iter__(self: Self) -> Self: ...
    def write(self, data: str) -> None: ...  # type: ignore[override]
-    def writelines(self, list: Iterable[str]) -> None: ...
+    def writelines(self, list: Iterable[str]) -> None: ...  # type: ignore[override]  # https://github.com/python/mypy/issues/14002


We probably don't need this type: ignore anymore (as well as several others), with the latest mypy release.

Cleaned up all comments referencing that issue.

AlexWaygood · 2023-02-08T08:45:24Z

stdlib/http/client.pyi

@@ -101,7 +101,7 @@ class HTTPMessage(email.message.Message):

 def parse_headers(fp: io.BufferedIOBase, _class: Callable[[], email.message.Message] = ...) -> HTTPMessage: ...

-class HTTPResponse(io.BufferedIOBase, BinaryIO):
+class HTTPResponse(io.BufferedIOBase, BinaryIO):  # type: ignore[misc]  # https://github.com/python/mypy/issues/14002


We do still need this type: ignore, but not because of the mypy issue linked (which is now closed) -- we need it because the definitions of writelines() in the two base classes are incompatible. The same goes for all the type: ignores in io.pyi and the one in lzma.pyi.

Updated these comments.

AlexWaygood · 2023-02-08T08:48:14Z

stdlib/tempfile.pyi

+    @overload
+    def write(self: _TemporaryFileWrapper[str], s: str) -> int: ...
+    @overload
+    def write(self: _TemporaryFileWrapper[bytes], s: ReadableBuffer) -> int: ...
+    @overload
+    def writelines(self: _TemporaryFileWrapper[str], lines: Iterable[str]) -> None: ...
+    @overload
+    def writelines(self: _TemporaryFileWrapper[bytes], lines: Iterable[ReadableBuffer]) -> None: ...


These overloads might not work very well with pyright, which has different handling of self types to mypy (#9591). The same goes for the overloads on lines 401-408, and the overloads in typing.pyi. You might want to instead do something similar to what I did with re.Pattern and re.Match in #9592.

Changed, and added some testcases for IO.write.

AlexWaygood · 2023-03-09T17:26:08Z

Ping @JelleZijlstra, have you had a chance to take a look at my review feedback? :)

JelleZijlstra · 2023-03-09T17:29:07Z

Oops, I think I assumed this was still blocked. I will take a look soon!

JelleZijlstra · 2023-03-10T03:58:23Z

hm, the new way to write the overloads makes subclassing much worse. Let me see if I can fix that.

github-actions · 2023-03-10T04:26:10Z

Diff from mypy_primer, showing the effect of this PR on open source code:

rich (https://github.com/Textualize/rich)
+ rich/progress.py:173: error: Definition of "writelines" in base class "IOBase" is incompatible with definition in base class "IO"  [misc]

psycopg (https://github.com/psycopg/psycopg)
+ psycopg/psycopg/copy.py:464: error: Unused "type: ignore" comment

cwltool (https://github.com/common-workflow-language/cwltool)
+ cwltool/main.py:963: error: Unused "type: ignore[arg-type]" comment

JelleZijlstra · 2023-03-10T04:29:58Z

Seems like we have a choice between:

One overload for AnyStr and another for Buffer: pyright accepts IO.write(AnyStr), but mypy hits Incorrect incompatible override report with self type + overload + TypeVar with constraints mypy#14866 and subclasses of IO classes that override the write/writelines method start complaining.
One overload for str and another for bytes + Buffer: mypy is fine with more subclasses, but pyright rejects IO.write(AnyStr).

I'm inclined to go with (1) because mypy's behavior is more clearly buggy than pyright's, but ideally we'd fix the mypy bug first.

AlexWaygood · 2023-03-12T11:55:57Z

Now that python/mypy#14882 has been merged, I guess we have two options:

we can either merge [alt] typing: accept buffers in IO.write #9861 now (and simplify the overloads after mypy 1.2 comes out, perhaps);
or we close [alt] typing: accept buffers in IO.write #9861, wait for mypy 1.2, and then merge this.

I'm easy either way :)

typing: accept buffers in typing.IO.write

2f87ea0

Fixes python#9082

JelleZijlstra mentioned this pull request Nov 4, 2022

Track review of bytes types #9006

Open

type ignore

e940cdf

This comment has been minimized.

Sign in to view

JelleZijlstra added 2 commits November 3, 2022 20:29

need this way

8d82f82

fix arg name

5aacf0a

This comment has been minimized.

Sign in to view

hauntsaninja added the status: deferred Issue or PR deferred until some precondition is fixed label Nov 5, 2022

Merge branch 'main' into typing

9293f5e

This comment has been minimized.

Sign in to view

AlexWaygood added 2 commits February 8, 2023 08:29

Merge branch 'main' into typing

717e864

Update lzma.pyi

2894148

This comment has been minimized.

Sign in to view

AlexWaygood removed the status: deferred Issue or PR deferred until some precondition is fixed label Feb 8, 2023

AlexWaygood reviewed Feb 8, 2023

View reviewed changes

Merge branch 'main' into typing

6532cd4

This comment has been minimized.

Sign in to view

JelleZijlstra and others added 7 commits March 9, 2023 19:10

Merge remote-tracking branch 'upstream/main' into typing

642f843

Remove references to closed mypy issue

641511b

Adjust overloads to work with pyright

590851b

[pre-commit.ci] auto fixes from pre-commit.com hooks

b6174cb

ignore another

92c2606

no picklebuffer in 3.7

7f10fc8

into the uncertain future

684b4ee

This comment has been minimized.

Sign in to view

JelleZijlstra requested a review from AlexWaygood March 10, 2023 03:55

maybe help mypy primer

bd78b5b

This comment has been minimized.

Sign in to view

JelleZijlstra mentioned this pull request Mar 10, 2023

Incorrect incompatible override report with self type + overload + TypeVar with constraints python/mypy#14866

Closed

try the other way

37bb27e

AlexWaygood mentioned this pull request Mar 10, 2023

[alt] typing: accept buffers in IO.write #9861

Merged

JelleZijlstra closed this in #9861 Mar 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

typing: accept buffers in typing.IO.write #9084

typing: accept buffers in typing.IO.write #9084

JelleZijlstra commented Nov 4, 2022

JelleZijlstra commented Nov 4, 2022

This comment has been minimized.

This comment has been minimized.

srittau commented Nov 4, 2022

JelleZijlstra commented Nov 4, 2022

hauntsaninja commented Nov 5, 2022

This comment has been minimized.

This comment has been minimized.

AlexWaygood left a comment

AlexWaygood Feb 8, 2023

JelleZijlstra Mar 10, 2023

AlexWaygood Feb 8, 2023

JelleZijlstra Mar 10, 2023

AlexWaygood Feb 8, 2023

JelleZijlstra Mar 10, 2023

This comment has been minimized.

AlexWaygood commented Mar 9, 2023

JelleZijlstra commented Mar 9, 2023

This comment has been minimized.

JelleZijlstra commented Mar 10, 2023

This comment has been minimized.

github-actions bot commented Mar 10, 2023

JelleZijlstra commented Mar 10, 2023

AlexWaygood commented Mar 12, 2023

typing: accept buffers in typing.IO.write #9084

typing: accept buffers in typing.IO.write #9084

Conversation

JelleZijlstra commented Nov 4, 2022

JelleZijlstra commented Nov 4, 2022

This comment has been minimized.

This comment has been minimized.

srittau commented Nov 4, 2022

JelleZijlstra commented Nov 4, 2022

hauntsaninja commented Nov 5, 2022

This comment has been minimized.

This comment has been minimized.

AlexWaygood left a comment

Choose a reason for hiding this comment

AlexWaygood Feb 8, 2023

Choose a reason for hiding this comment

JelleZijlstra Mar 10, 2023

Choose a reason for hiding this comment

AlexWaygood Feb 8, 2023

Choose a reason for hiding this comment

JelleZijlstra Mar 10, 2023

Choose a reason for hiding this comment

AlexWaygood Feb 8, 2023

Choose a reason for hiding this comment

JelleZijlstra Mar 10, 2023

Choose a reason for hiding this comment

This comment has been minimized.

AlexWaygood commented Mar 9, 2023

JelleZijlstra commented Mar 9, 2023

This comment has been minimized.

JelleZijlstra commented Mar 10, 2023

This comment has been minimized.

github-actions bot commented Mar 10, 2023

JelleZijlstra commented Mar 10, 2023

AlexWaygood commented Mar 12, 2023