isort does not recognize _collections_abc as part of the standard library

[LLyaudet]

Hello,

In my repository:
https://github.com/LLyaudet/python-repeatable-iterable/blob/main/src/python_repeatable_iterable/__init__.py
I have the following imports:

from typing import Iterable, Iterator, TypeVar, cast
from _collections_abc import dict_items, dict_keys, dict_values

from python_none_objects import NoneIterable

But isort does not recognize that

from _collections_abc import dict_items, dict_keys, dict_values

is part of the standard library and regroup it like that:

from typing import Iterable, Iterator, TypeVar, cast

from _collections_abc import dict_items, dict_keys, dict_values
from python_none_objects import NoneIterable

Best regards,
Laurent Lyaudet

[devdanzin]

Hello, we would like to fix this issue. Is there agreement that _collections_abc should be added to the list of standard library modules?

[LLyaudet]

Hello @devdanzin :),
Thanks for looking at this problem.
Sorry for my late answer, I have been away from my laptop for some time.
There is indeed this agreement.
_collections_abc is somewhat a private standard library modules, as is shown by this URL :

https://github.com/python/cpython/blob/main/Lib/_collections_abc.py

I would suggest that you add all other "private standard library modules" that are listed here:
https://github.com/python/cpython/tree/main/Lib

Thank you very much, best regards,
Laurent Lyaudet

[LLyaudet]

Hello @devdanzin, @mtr-d3v, :),
https://docs.python.org/3/library/index.html
states that built-in modules are stdlib modules that are written in C contrary to other Python stdlib modules that are written in Python.

The library contains built-in modules (written in C) that provide access to system functionality such as file I/O that would otherwise be inaccessible to Python programmers, as well as modules written in Python that provide standardized solutions for many problems that occur in everyday programming.

Thus, there may be confusion with builtin functions, exceptions, and constants that can be used in Python code without import, but the builtin module sys for example still requires an import sys to be used.

A complete list can be given by sys.stdlib_module_names.

laurent@laurent-GL73-8SD:~$ python3
Python 3.12.3 (main, Jul 31 2024, 17:43:48) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.builtin_module_names
('_abc', '_ast', '_bisect', '_blake2', '_codecs', '_collections', '_csv', '_datetime', '_elementtree', '_functools', '_heapq', '_imp', '_io', '_locale', '_md5', '_opcode', '_operator', '_pickle', '_posixsubprocess', '_random', '_sha1', '_sha2', '_sha3', '_signal', '_socket', '_sre', '_stat', '_statistics', '_string', '_struct', '_symtable', '_thread', '_tokenize', '_tracemalloc', '_typing', '_warnings', '_weakref', 'array', 'atexit', 'binascii', 'builtins', 'cmath', 'errno', 'faulthandler', 'fcntl', 'gc', 'grp', 'itertools', 'marshal', 'math', 'posix', 'pwd', 'pyexpat', 'select', 'spwd', 'sys', 'syslog', 'time', 'unicodedata', 'zlib')
>>> sys.stdlib_module_names
frozenset({'_osx_support', 'tracemalloc', 'doctest', 'cmd', 'codecs', '_codecs_jp', 'telnetlib', 'warnings', '_lzma', '_codecs_hk', '_contextvars', '_collections_abc', '_sqlite3', 'code', '_threading_local', 'gettext', '_csv', 'mailcap', 'queue', 'selectors', '_overlapped', 'pipes', 'zlib', 'dbm', 'chunk', '_tracemalloc', 'cgi', 'array', 'pdb', 'turtle', 'copyreg', 'cgitb', 'colorsys', 'signal', '_multiprocessing', '_ast', 'curses', 'grp', 'mailbox', 'ipaddress', 'fnmatch', 'wave', 'tty', 'imghdr', '_bz2', 'sre_parse', 'gc', '_compression', 'html', 'pydoc_data', 'typing', '_locale', 'io', 'contextlib', '__future__', 'filecmp', 'runpy', 'string', 'unicodedata', 'pathlib', 'turtledemo', 'ntpath', 'sqlite3', 'pty', '_msi', 'concurrent', '_asyncio', 'weakref', '_struct', '_thread', '_zoneinfo', 'msilib', '_uuid', '_curses_panel', 'pstats', 'nt', 'sched', '_weakref', '_codecs_kr', '_codecs_tw', 'sre_constants', 'linecache', 'pyclbr', 'atexit', '_sha3', 'idlelib', 'termios', 'sysconfig', 'modulefinder', 'tokenize', 'codeop', 'unittest', 'sre_compile', 'smtplib', '_markupbase', 'base64', 'encodings', 'fractions', '_io', 'multiprocessing', 'getpass', 'subprocess', 'pyexpat', 'importlib', '_abc', 'wsgiref', 'hashlib', 'select', 'rlcompleter', 'audioop', 'this', 'antigravity', 'pkgutil', '_decimal', '_gdbm', 'lzma', 'fcntl', '_signal', 'numbers', 'json', 'platform', 'ctypes', 'getopt', '_sha2', 'genericpath', 'plistlib', 'dataclasses', 'pwd', 'tempfile', 'resource', '_dbm', 'winsound', 'zoneinfo', '_random', 'reprlib', 'itertools', '_statistics', 'math', '_hashlib', 'decimal', 'zipfile', 'nturl2path', '_bisect', '_posixshmem', '_scproxy', 'venv', 'operator', 'hmac', 'errno', 'textwrap', 'readline', 'asyncio', '_multibytecodec', 'posixpath', 'zipimport', '_tokenize', 'sys', 'keyword', 'binascii', 'configparser', 'shutil', '_lsprof', 'stringprep', 'ast', 'contextvars', 'shelve', 'crypt', 'nis', 'profile', 'stat', '_string', '_opcode', 'spwd', '_pylong', 'winreg', '_aix_support', '_socket', '_sha1', '_pyio', 'cProfile', '_py_abc', 'logging', '_codecs_iso2022', 'marshal', 'struct', 'symtable', '_codecs', '_codecs_cn', 'os', 'graphlib', '_sre', 'xml', '_posixsubprocess', 'urllib', '_operator', 'nntplib', 'enum', '_curses', 'builtins', 'csv', '_tkinter', 'tkinter', 'quopri', 'tarfile', 'mmap', 'ossaudiodev', 'pickletools', 'sndhdr', '_queue', 'secrets', '_blake2', 'xmlrpc', '_compat_pickle', '_collections', 'threading', 'webbrowser', 'pickle', '_frozen_importlib', 'copy', '_md5', '_functools', '_elementtree', 'gzip', 're', 'timeit', 'statistics', 'shlex', 'zipapp', '_ctypes', 'poplib', '_symtable', 'py_compile', 'ssl', 'ensurepip', 'imaplib', 'bz2', 'bisect', 'opcode', '_strptime', 'posix', 'functools', 'calendar', 'socket', 'socketserver', 'xdrlib', 'random', 'msvcrt', 'netrc', 'argparse', 'optparse', 'http', 'mimetypes', '_stat', '_json', 'syslog', 'inspect', 'datetime', '_crypt', 'dis', 'token', 'collections', 'faulthandler', '_pydecimal', 'uu', 'uuid', 'tabnanny', 'pydoc', '_frozen_importlib_external', '_pydatetime', '_heapq', 'email', 'sunau', '_weakrefset', 'difflib', 'compileall', 'trace', 'types', 'glob', '_ssl', '_warnings', 'ftplib', '_sitebuiltins', 'fileinput', 'aifc', 'time', 'cmath', 'abc', 'heapq', 'lib2to3', 'site', '_pickle', 'tomllib', 'locale', '_typing', '_winapi', 'bdb', 'traceback', '_imp', 'pprint', '_datetime'})

Note that sys.stdlib_module_names has been added in Python 3.10.
python/cpython#87121
Before that only sys.builtin_module_names was available.

But unfortunately, isort code use the documentation to know the modules list:
https://www.sphinx-doc.org/en/master/usage/domains/python.html
py:module
https://github.com/PyCQA/isort/blob/main/scripts/mkstdlibs.py

As of now I don't have yet a way to fetch all stdlib modules for a specific Python version using sphinx.
The only solution I see right now is to launch a docker image using Python images on Dockerhub.
https://hub.docker.com/_/python
It would work with sys.stdlib_module_names for Python 3.10 and after.
And it would work partially for Python 3.9 and before.

Maybe a sphinx specialist would be able to find a way?

Best regards,
Laurent Lyaudet

[devdanzin]

Hi @LLyaudet, sorry for my late reply, I've also been away for a while.

I've created #2295 implementing your idea of using sys.stdlib_module_names for Python 3.10 and later, falling back to sys.builtin_module_names in 3.8 and 3.9. It still downloads the module list using Sphinx and uses the information from sys to complement the fetched list, which is important for these older versions.

This fixes your original report and many other omissions, but adds a lot of new entries to the known modules lists so isort developers should assess whether there is any downside to this approach.

Thanks for your analysis and pointers!

[LLyaudet]

Hello @devdanzin :),
Nice, thank you :).
Best regards,
Laurent