Infinite recursion in isinstance check

[ksunden]

A repo that reproduces this issue (with CI checks running) can be found at ksunden/typing_recursion

This appears to be a resurgence of #503 / #501, but in a very particular set of circumstances:

• In a weakref.finalize call
• In a relative import
• When typing is imported
• Using pytest via setup.py test
• only if the test script is not in the module
• Python <=3.6 (have not tested 3.4 and below, have tested on 3.7)

The crux of the issue seems to boil down two seemingly identical entries in cls.__extra__.subclasses__() (I've seen it as either typing.Generator or typing.ContextManager, depending on what isinstance check causes it.)

However, one of the two entries fails the check isinstance(scls, GenericMeta), despite the fact that it's parent class is, in fact typing.GenericMeta by any other method of inspection.

I'm not actually convinced this is the fault of typing.py, could be either setuptools or pytest or even weakref, but the issue has been seen in this repo before, so starting here.

Note that the issue presents after CI checks pass, so particularly hard to test for properly.

Particularly relevant lines of code:

https://github.com/python/typing/blob/master/src/typing.py#L882-L885

[ksunden]

Potentially related to #562, though doesn't look to necessarily be happening in the same part of the code, so not sure

[ilevkivskyi]

Thanks for reporting! This looks like a bizarre bug (also you have made a great analysis).

Have you tried replacing the isinstance(scls, GenericMeta) check you mention with GenericMeta in type(scls).__mro__? Do you still see the crash? Are other typing test passing with this change?

[ksunden]

The problem persists with GenericMeta in type(scls).__mro__

Wondering if there is some form of monkeypatching going on, but have as yet been unable to pin down where that would be happening.

[ilevkivskyi]

Another idea is just to add a debugging print to see what is type(cls).__mro__ at t hat point for each scls. Maybe this will clarify something (I could of course do this myself, but really don't have time for this sorry.)

[ksunden]

I'll get back to actively debugging when I get the chance.
I did print type(scls) at one point during my original debugging run. Both entries say that they are typing.GenericMeta, though isinstance still fails.

I was able to (crudely) fix the problem by doing string comparison on type(scls), but that does not seem like a robust long term solution.

[ahitrin]

We have similar issue in our project (running Python 2.7.15) when there is more than 1 instance of typing library in module path (this is caused by our custom hacks with dependency packaging).

This allows me to find a very simple way to reproduce a bug. You just need to import a library twice, and two versions will fight against each other:

$ cp python2/typing.py python2/typing2.py
$ cat reproduce.py
import typing
import typing2
from collections import Mapping
assert isinstance(1, Mapping) is False
$ PYTHONPATH=python2 reproduce.py
Traceback (most recent call last):
  File "reproduce.py", line 4, in <module>
    assert isinstance(1, Mapping) is False
  File "/home/ahitrin/src/typing/.env/lib/python2.7/abc.py", line 144, in __instancecheck__
    return cls.__subclasscheck__(subtype)
  File "/home/ahitrin/src/typing/.env/lib/python2.7/abc.py", line 180, in __subclasscheck__
    if issubclass(subclass, scls):
  File "/home/ahitrin/src/typing/.env/lib/python2.7/abc.py", line 180, in __subclasscheck__
    if issubclass(subclass, scls):
  File "/home/ahitrin/src/typing/python2/typing.py", line 1250, in __subclasscheck__
    return super(GenericMeta, self).__subclasscheck__(cls)
  File "/home/ahitrin/src/typing/.env/lib/python2.7/abc.py", line 161, in __subclasscheck__
    ok = cls.__subclasshook__(subclass)
  File "/home/ahitrin/src/typing/python2/typing.py", line 982, in __extrahook__
    if issubclass(subclass, scls):
  File "/home/ahitrin/src/typing/python2/typing2.py", line 1250, in __subclasscheck__
    return super(GenericMeta, self).__subclasscheck__(cls)
  File "/home/ahitrin/src/typing/.env/lib/python2.7/abc.py", line 161, in __subclasscheck__
    ok = cls.__subclasshook__(subclass)
  File "/home/ahitrin/src/typing/python2/typing2.py", line 982, in __extrahook__
    if issubclass(subclass, scls):
  File "/home/ahitrin/src/typing/python2/typing.py", line 1250, in __subclasscheck__
    return super(GenericMeta, self).__subclasscheck__(cls)
  File "/home/ahitrin/src/typing/.env/lib/python2.7/abc.py", line 161, in __subclasscheck__
    ok = cls.__subclasshook__(subclass)
  File "/home/ahitrin/src/typing/python2/typing.py", line 982, in __extrahook__
    if issubclass(subclass, scls):
  File "/home/ahitrin/src/typing/python2/typing2.py", line 1250, in __subclasscheck__
    return super(GenericMeta, self).__subclasscheck__(cls)
  File "/home/ahitrin/src/typing/.env/lib/python2.7/abc.py", line 161, in __subclasscheck__
    ok = cls.__subclasshook__(subclass)
  File "/home/ahitrin/src/typing/python2/typing2.py", line 982, in __extrahook__
    if issubclass(subclass, scls):
  File "/home/ahitrin/src/typing/python2/typing.py", line 1250, in __subclasscheck__
    return super(GenericMeta, self).__subclasscheck__(cls)
  File "/home/ahitrin/src/typing/.env/lib/python2.7/abc.py", line 161, in __subclasscheck__
    ok = cls.__subclasshook__(subclass)
   ...
  File "/home/ahitrin/src/typing/python2/typing2.py", line 982, in __extrahook__
    if issubclass(subclass, scls):
  File "/home/ahitrin/src/typing/python2/typing.py", line 1250, in __subclasscheck__
    return super(GenericMeta, self).__subclasscheck__(cls)
  File "/home/ahitrin/src/typing/.env/lib/python2.7/abc.py", line 161, in __subclasscheck__
    ok = cls.__subclasshook__(subclass)
  File "/home/ahitrin/src/typing/python2/typing.py", line 982, in __extrahook__
    if issubclass(subclass, scls):
  File "/home/ahitrin/src/typing/python2/typing2.py", line 1250, in __subclasscheck__
    return super(GenericMeta, self).__subclasscheck__(cls)
  File "/home/ahitrin/src/typing/.env/lib/python2.7/abc.py", line 161, in __subclasscheck__
    ok = cls.__subclasshook__(subclass)
  File "/home/ahitrin/src/typing/python2/typing2.py", line 982, in __extrahook__
    if issubclass(subclass, scls):
  File "/home/ahitrin/src/typing/python2/typing.py", line 1250, in __subclasscheck__
    return super(GenericMeta, self).__subclasscheck__(cls)
  File "/home/ahitrin/src/typing/.env/lib/python2.7/abc.py", line 161, in __subclasscheck__
    ok = cls.__subclasshook__(subclass)
  File "/home/ahitrin/src/typing/python2/typing.py", line 982, in __extrahook__
    if issubclass(subclass, scls):
  File "/home/ahitrin/src/typing/python2/typing2.py", line 1250, in __subclasscheck__
    return super(GenericMeta, self).__subclasscheck__(cls)
  File "/home/ahitrin/src/typing/.env/lib/python2.7/abc.py", line 161, in __subclasscheck__
    ok = cls.__subclasshook__(subclass)
  File "/home/ahitrin/src/typing/python2/typing2.py", line 982, in __extrahook__
    if issubclass(subclass, scls):
  File "/home/ahitrin/src/typing/python2/typing.py", line 1250, in __subclasscheck__
    return super(GenericMeta, self).__subclasscheck__(cls)
  File "/home/ahitrin/src/typing/.env/lib/python2.7/abc.py", line 151, in __subclasscheck__
    if subclass in cls._abc_cache:
RuntimeError: maximum recursion depth exceeded while calling a Python object

This script works both for py2 and py3 versions (at least on 3.6).

The similar stack trace for Py3:

$ PYTHONPATH=src .3env/bin/python reproduce.py                                                                                                                                                                     
Traceback (most recent call last):
  File "reproduce.py", line 4, in <module>
    assert isinstance(1, Mapping) is False
  File "/home/ahitrin/src/typing/.3env/lib/python3.6/abc.py", line 193, in __instancecheck__
    return cls.__subclasscheck__(subclass)
  File "/home/ahitrin/src/typing/.3env/lib/python3.6/abc.py", line 228, in __subclasscheck__
    if issubclass(subclass, scls):
  File "/home/ahitrin/src/typing/.3env/lib/python3.6/abc.py", line 228, in __subclasscheck__
    if issubclass(subclass, scls):
  File "/home/ahitrin/src/typing/.3env/lib/python3.6/abc.py", line 228, in __subclasscheck__
    if issubclass(subclass, scls):
  File "/home/ahitrin/src/typing/src/typing.py", line 1155, in __subclasscheck__
    return super().__subclasscheck__(cls)
  File "/home/ahitrin/src/typing/.3env/lib/python3.6/abc.py", line 209, in __subclasscheck__
    ok = cls.__subclasshook__(subclass)
  File "/home/ahitrin/src/typing/src/typing.py", line 885, in __extrahook__
    if issubclass(subclass, scls):
  File "/home/ahitrin/src/typing/src/typing2.py", line 1155, in __subclasscheck__
    return super().__subclasscheck__(cls)
  File "/home/ahitrin/src/typing/.3env/lib/python3.6/abc.py", line 209, in __subclasscheck__
    ok = cls.__subclasshook__(subclass)
  File "/home/ahitrin/src/typing/src/typing2.py", line 885, in __extrahook__
    if issubclass(subclass, scls):
  File "/home/ahitrin/src/typing/src/typing.py", line 1155, in __subclasscheck__
    return super().__subclasscheck__(cls)
...
  File "/home/ahitrin/src/typing/src/typing2.py", line 885, in __extrahook__
    if issubclass(subclass, scls):
  File "/home/ahitrin/src/typing/src/typing.py", line 1155, in __subclasscheck__
    return super().__subclasscheck__(cls)
  File "/home/ahitrin/src/typing/.3env/lib/python3.6/abc.py", line 209, in __subclasscheck__
    ok = cls.__subclasshook__(subclass)
  File "/home/ahitrin/src/typing/src/typing.py", line 885, in __extrahook__
    if issubclass(subclass, scls):
  File "/home/ahitrin/src/typing/src/typing2.py", line 1155, in __subclasscheck__
    return super().__subclasscheck__(cls)
  File "/home/ahitrin/src/typing/.3env/lib/python3.6/abc.py", line 209, in __subclasscheck__
    ok = cls.__subclasshook__(subclass)
  File "/home/ahitrin/src/typing/src/typing2.py", line 885, in __extrahook__
    if issubclass(subclass, scls):
  File "/home/ahitrin/src/typing/src/typing.py", line 1155, in __subclasscheck__
    return super().__subclasscheck__(cls)
  File "/home/ahitrin/src/typing/.3env/lib/python3.6/abc.py", line 209, in __subclasscheck__
    ok = cls.__subclasshook__(subclass)
  File "/home/ahitrin/src/typing/src/typing.py", line 885, in __extrahook__
    if issubclass(subclass, scls):
  File "/home/ahitrin/src/typing/src/typing2.py", line 1155, in __subclasscheck__
    return super().__subclasscheck__(cls)
  File "/home/ahitrin/src/typing/.3env/lib/python3.6/abc.py", line 199, in __subclasscheck__
    if subclass in cls._abc_cache:
  File "/home/ahitrin/src/typing/.3env/lib/python3.6/_weakrefset.py", line 72, in __contains__
    wr = ref(item)
RecursionError: maximum recursion depth exceeded while calling a Python object

[ilevkivskyi]

This allows me to find a very simple way to reproduce a bug. You just need to import a library twice, and two versions will fight against each other

Great, thanks! I just tried and this is fixed in Python 3.7. I think I understand the cause of the crash, the problem is that there are two instances of GenericMeta, and the isinstance(scls, GenericMeta) fails in one module for the subclass created in the other module. This would be therefore tricky to fix.

[srittau]

As the typing module is not part of Python, please file a bug report at bugs.python.org if this is still an issue.

[herronelou]

Is there anything one could do to prevent this situation from happening?
I'm running in a situation where my project has a dependency on multiple external packages, 2 of which provide their own copy of typing (one as typing, one as typing27), and the same infinite recursion as @ahitrin happens.
I could possibly patch up one of them ('typing') while the other one comes bundled up with the base application that runs my program and is nearly untouchable.