ModuleNotFoundError: No module named 'distutils' in latest release

[rickgamez]

Expected behaviour

distutils is deprecated and should not be used in the module.

Actual behaviour

Installing latest version of opencv_python runs into an exception because of the distutils reference.

Call Stack

ERROR: Exception:
Traceback (most recent call last):
File "...Python312\Lib\site-packages\pip_internal\cli\base_command.py", line 180, in exc_logging_wrapper
status = run_func(*args)
^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\cli\req_command.py", line 245, in wrapper
return func(self, options, args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\commands\install.py", line 377, in run
requirement_set = resolver.resolve(
^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\resolution\resolvelib\resolver.py", line 95, in resolve
result = self._result = resolver.resolve(
^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_vendor\resolvelib\resolvers.py", line 546, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_vendor\resolvelib\resolvers.py", line 397, in resolve
self._add_to_criteria(self.state.criteria, r, parent=None)
File "...Python312\Lib\site-packages\pip_vendor\resolvelib\resolvers.py", line 173, in _add_to_criteria
if not criterion.candidates:
^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_vendor\resolvelib\structs.py", line 156, in bool
return bool(self._sequence)
^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\resolution\resolvelib\found_candidates.py", line 155, in bool
return any(self)
^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\resolution\resolvelib\found_candidates.py", line 143, in
return (c for c in iterator if id(c) not in self._incompatible_ids)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\resolution\resolvelib\found_candidates.py", line 97, in _iter_built_with_inserted
candidate = func()
^^^^^^
File "...Python312\Lib\site-packages\pip_internal\resolution\resolvelib\factory.py", line 182, in _make_candidate_from_link
base: Optional[BaseCandidate] = self._make_base_candidate_from_link(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\resolution\resolvelib\factory.py", line 228, in _make_base_candidate_from_link
self._link_candidate_cache[link] = LinkCandidate(
^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\resolution\resolvelib\candidates.py", line 290, in init
super().init(
File "...Python312\Lib\site-packages\pip_internal\resolution\resolvelib\candidates.py", line 156, in init
self.dist = self._prepare()
^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\resolution\resolvelib\candidates.py", line 222, in _prepare
dist = self._prepare_distribution()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\resolution\resolvelib\candidates.py", line 301, in _prepare_distribution
return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\operations\prepare.py", line 525, in prepare_linked_requirement
return self._prepare_linked_requirement(req, parallel_builds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\operations\prepare.py", line 640, in _prepare_linked_requirement
dist = _get_prepared_distribution(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\operations\prepare.py", line 71, in _get_prepared_distribution
abstract_dist.prepare_distribution_metadata(
File "...Python312\Lib\site-packages\pip_internal\distributions\sdist.py", line 54, in prepare_distribution_metadata
self._install_build_reqs(finder)
File "...Python312\Lib\site-packages\pip_internal\distributions\sdist.py", line 124, in _install_build_reqs
build_reqs = self._get_build_requires_wheel()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\distributions\sdist.py", line 101, in _get_build_requires_wheel
return backend.get_requires_for_build_wheel()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_internal\utils\misc.py", line 745, in get_requires_for_build_wheel
return super().get_requires_for_build_wheel(config_settings=cs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_vendor\pyproject_hooks_impl.py", line 166, in get_requires_for_build_wheel
return self._call_hook('get_requires_for_build_wheel', {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\site-packages\pip_vendor\pyproject_hooks_impl.py", line 321, in _call_hook
raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):
File "...Python312\Lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 77, in build_backend
obj = import_module(mod_path)
^^^^^^^^^^^^^^^^^^^^^^^
File "...Python312\Lib\importlib_init.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1310, in _find_and_load_unlocked
File "", line 488, in _call_with_frames_removed
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1331, in _find_and_load_unlocked
File "", line 935, in _load_unlocked
File "", line 995, in exec_module
File "", line 488, in call_with_frames_removed
File "...overlay\Lib\site-packages\setuptools_init.py", line 10, in
import distutils.core
ModuleNotFoundError: No module named 'distutils'

[adamjstewart]

Same issue in our pipelines: https://github.com/microsoft/torchgeo/actions/runs/9356814161/job/25755818665?pr=2103

[dbrnz]

Same issue here...

[johnthagen]

distutils was removed in Python 3.12

• https://docs.python.org/3/whatsnew/3.12.html#summary-release-highlights

[dbrnz]

Same issue here...

The real issue is #994.

[adamjstewart]

Would also be good to get the source build working in Python 3.12+ even if macOS arm64 wheels are available. May be as simple as removing the pin to older setuptools versions?

[johnthagen]

MacOS ARM wheels are now available, but as mentioned the sdist should probably still be fixed

• MacOS arm64 wheels missing from PyPI 4.10 release #994 (comment)

[1riggs]

Related to this issue, the python:alpine image includes the updated setuptools which resolves the missing distutils module but installation of opencv-python still fails.

For context, trying to build my own package which includes opencv-python as a dependency, but I am having issues installing opencv-python using pip in alpine 3.20 (python3.12).

Steps to reproduce using simplified dockerfile and no source code

FROM python:alpine

RUN pip3 install opencv-python

then:

$ docker build -t opencv-python .
[+] Building 10.8s (6/6) FINISHED                                                                                                                                                                   docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                          0.0s
 => => transferring dockerfile: 131B                                                                                                                                                                          0.0s
 => [internal] load metadata for docker.io/library/python:alpine                                                                                                                                              0.0s
 => [internal] load .dockerignore                                                                                                                                                                             0.0s
 => => transferring context: 2B                                                                                                                                                                               0.0s
 => [1/3] FROM docker.io/library/python:alpine                                                                                                                                                                0.0s
 => CACHED [2/3] RUN pip3 install --upgrade setuptools pip                                                                                                                                                    0.0s
 => ERROR [3/3] RUN pip3 install opencv-python                                                                                                                                                               10.8s
------                                                                                                                                                                                                             
 > [3/3] RUN pip3 install opencv-python:                                                                                                                                                                           
0.669 Collecting opencv-python                                                                                                                                                                                     
0.939   Downloading opencv-python-4.10.0.84.tar.gz (95.1 MB)                                                                                                                                                       
3.129      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 95.1/95.1 MB 27.7 MB/s eta 0:00:00                                                                                                                             
5.320   Installing build dependencies: started                                                                                                                                                                     
10.43   Installing build dependencies: finished with status 'done'
10.44   Getting requirements to build wheel: started
10.51   Getting requirements to build wheel: finished with status 'error'
10.51   error: subprocess-exited-with-error
10.51   
10.51   × Getting requirements to build wheel did not run successfully.
10.51   │ exit code: 1
10.51   ╰─> [33 lines of output]
10.51       Traceback (most recent call last):
10.51         File "/usr/local/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
10.51           main()
10.51         File "/usr/local/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
10.51           json_out['return_val'] = hook(**hook_input['kwargs'])
10.51                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
10.51         File "/usr/local/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 112, in get_requires_for_build_wheel
10.51           backend = _build_backend()
10.51                     ^^^^^^^^^^^^^^^^
10.51         File "/usr/local/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
10.51           obj = import_module(mod_path)
10.51                 ^^^^^^^^^^^^^^^^^^^^^^^
10.51         File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
10.51           return _bootstrap._gcd_import(name[level:], package, level)
10.51                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
10.51         File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
10.51         File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
10.51         File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
10.51         File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
10.51         File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
10.51         File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
10.51         File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
10.51         File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
10.51         File "<frozen importlib._bootstrap_external>", line 995, in exec_module
10.51         File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
10.51         File "/tmp/pip-build-env-rzx3tyxe/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 16, in <module>
10.51           import setuptools.version
10.51         File "/tmp/pip-build-env-rzx3tyxe/overlay/lib/python3.12/site-packages/setuptools/version.py", line 1, in <module>
10.51           import pkg_resources
10.51         File "/tmp/pip-build-env-rzx3tyxe/overlay/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2172, in <module>
10.51           register_finder(pkgutil.ImpImporter, find_on_path)
10.51                           ^^^^^^^^^^^^^^^^^^^
10.51       AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
10.51       [end of output]
10.51   
10.51   note: This error originates from a subprocess, and is likely not a problem with pip.
10.51 error: subprocess-exited-with-error
10.51 
10.51 × Getting requirements to build wheel did not run successfully.
10.51 │ exit code: 1
10.51 ╰─> See above for output.
10.51 
10.51 note: This error originates from a subprocess, and is likely not a problem with pip.
------
Dockerfile:4
--------------------
   2 |     
   3 |     RUN pip3 install --upgrade setuptools pip
   4 | >>> RUN pip3 install opencv-python
   5 |     
   6 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c pip3 install opencv-python" did not complete successfully: exit code: 1

A possible workaround is to install opencv-python from the alpine repository

FROM python:alpine

RUN apk add py3-opencv

but the package is not available to the runtime python distribution (not included in the output of pip list).

$ docker run --rm -it opencv-python pip list
Package    Version
---------- -------
pip        24.0
setuptools 70.3.0
wheel      0.43.0

[Avasam]

@rickgamez

File "...Python312\Lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 77, in build_backend
[...]
File "...Python312\Lib\importlib_init.py", line 90, in import_module
[...]
File "...overlay\Lib\site-packages\setuptools_init.py", line 10, in
import distutils.core
ModuleNotFoundError: No module named 'distutils'

Have you tried updating pip, it used to vendor setuptools, which is of course an outdated vendor.

---

Same issue in our pipelines: microsoft/torchgeo/actions/runs/9356814161/job/25755818665?pr=2103

setuptools/init.py", line 10, in
import distutils.core
ModuleNotFoundError: No module named 'distutils'

@adamjstewart

Have you tried with the environment variable SETUPTOOLS_USE_DISTUTILS NOT set to stdlib (the other intended valid value is local, but the implementation details work in a way that any value that's not stdlib should cause to prefer using the vendored version)

Have you tried updating setuptools ? I think it should fallback to its vendored version even with SETUPTOOLS_USE_DISTUTILS=stdlib

In your own code, make sure you always import setuptools before distutils.

---

@1riggs Lastly, this PR should also be merged: #1012 , see pypa/setuptools#3935 (comment)

[adamjstewart]

Supporting newer setuptools is great, but is there a reason we don't just remove all references to distutils? The setuptools copy of distutils is also planned to be deprecated, so we're just kicking the can down the road instead of solving the root problem.

[adamjstewart]

Oh wait, opencv doesn't use distutils, setuptools does. So yes, I think supporting newer setuptools really is the solution here. Thanks for the fix @bryankaplan and the pointer @Avasam!

[CanePlayz]

I've also encountered this issue today when trying to install opencv on a Windows arm64 machine. So either the build process should be fixed or we could also supply wheels for Windows arm64 architecture.

[bryankaplan]

@CanePlayz I opened #1012 almost four months ago, and it's just sitting there unmerged. Someone reported that it didn't work under MacOS. Can you please test whether it works on your Windows arm64 machine?

[CanePlayz]

Sure, I'll check tomorrow. Since I'm new to this, just to make sure: I'll just have to run python setup.py bdist_wheel to build the package locally?

[bryankaplan]

@CanePlayz Negative; see item 5 under Manual Builds.

[CanePlayz]

Ok, thanks.