Pickle ignores custom getstate methods on TextIOWrapper in Python 3.12

[SimonSorgQC]

Bug report

Bug description:

So I am not entirely sure whether this is unintended behaviour, but it is definitely a noticeable change between 3.11 and 3.12 that is rather unintuitive

import pickle
from io import BytesIO, TextIOWrapper

class EncodedFile(TextIOWrapper):
	def __getstate__(self):
		return "string"
	def __setstate__(self, state):
		pass

file = EncodedFile(BytesIO(b"string"))
pickle.dumps(file)

This works in Python 3.11 and 3.10, but fails in 3.12 with

pickle.dumps(file)
TypeError: cannot pickle 'EncodedFile' instances

CPython versions tested on:

3.10, 3.11, 3.12

Operating systems tested on:

macOS

Linked PRs

• gh-122559: Synchronize C and Python implementation of the io module about pickling #122628

[tomasr8]

Seems like it was added in this PR: https://github.com/python/cpython/pull/101948/files#diff-f6d1380274f6859688c821fced58fb1c829d8acad44ae252ba988e220e1080b3R3240

[SimonSorgQC]

Interesting! While it makes sense that the original IOBase can not be pickled, I feel like it should be possible for subclasses overriding getstate

[SimonSorgQC]

So I can verify that overriding reduce and reduce_ex worked for me. I would leave the issue open for now as I do not think getstate and setstate not working is nice - especially since reduce and reduce_ex of IOBase are buried in the C code.

Please feel free to correct me if you think this has been resolved.

[serhiy-storchaka]

Pickling file objects was explicitly forbidden in bpo-10180 by adding __getstate__ methods that raise TypeError. These methods were removed from the C implementation in bpo-33138, because these classes were recognized as non-pickleable by default. Then these classes were made heap types in #101948 (gh-101819), and new __reduce__ and __reduce_ex__ methods that raise TypeError were added, because condition that prevented them from pickling was gone.

The right way is to remove __reduce__ and __reduce_ex__ methods and restore __getstate__ methods. This will make the C implementation consistent with the Python implementation.

[serhiy-storchaka]

It turned out that this issue is a bit more complex. There are more differences between implementations. Classes are not tested separately, existing tests do not cover all cases. I am working on this problem.

[serhiy-storchaka]

I added tests as a guard against such type of regression, but in general there are no guaranties about pickleability (especially with protocols 0 and 1).

In meantime, the workaround is to reset __reduce__ and __reduce_ex__ in your subclass to default:

class EncodedFile(TextIOWrapper):
    ...
    __reduce__ = object.__reduce__
    __reduce_ex__ = object.__reduce_ex__