[bug]: venv creation fails on linux-musl builds

[yozachar]

I installed a standalone build with mise, but I'm unable to create virtual environments.

Error

$ python -m venv .venv
Error: Command '['/home/dan/check/.venv/bin/python', '-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

Info

$ python
Python 3.12.3 (main, Apr 15 2024, 18:29:24) [Clang 14.0.3 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
$ which python
/home/dan/.local/share/mise/installs/python/latest/bin/python
$ which pip
/home/dan/.local/share/mise/installs/python/latest/bin/pip
$ pip list
Package    Version
---------- -------
pip        24.0
setuptools 69.1.0
$ pip debug
WARNING: This command is only meant for debugging. Do not use this with automation for parsing and getting these details, since the output and options of this command may change without notice.
pip version: pip 24.0 from /home/dan/.local/share/mise/installs/python/latest/lib/python3.12/site-packages/pip (python 3.12)
sys.version: 3.12.3 (main, Apr 15 2024, 18:29:24) [Clang 14.0.3 ]
sys.executable: /home/dan/.local/share/mise/installs/python/latest/bin/python3.12
sys.getdefaultencoding: utf-8
sys.getfilesystemencoding: utf-8
locale.getpreferredencoding: UTF-8
sys.platform: linux
sys.implementation:
  name: cpython
'cert' config value: Not specified
REQUESTS_CA_BUNDLE: None
CURL_CA_BUNDLE: None
pip._vendor.certifi.where(): /home/dan/.local/share/mise/installs/python/latest/lib/python3.12/site-packages/pip/_vendor/certifi/cacert.pem
pip._vendor.DEBUNDLED: False
vendored library versions:
  CacheControl==0.13.1
  colorama==0.4.6
  distlib==0.3.8
  distro==1.8.0
  msgpack==1.0.5
  packaging==21.3
  platformdirs==3.8.1
  pyparsing==3.1.0
  pyproject-hooks==1.0.0
  requests==2.31.0
  certifi==2023.07.22
  chardet==5.1.0
  idna==3.4
  urllib3==1.26.17
  rich==13.4.2 (Unable to locate actual module version, using vendor.txt specified version)
  pygments==2.15.1
  typing_extensions==4.7.1 (Unable to locate actual module version, using vendor.txt specified version)
  resolvelib==1.0.1
  setuptools==68.0.0 (Unable to locate actual module version, using vendor.txt specified version)
  six==1.16.0
  tenacity==8.2.2 (Unable to locate actual module version, using vendor.txt specified version)
  tomli==2.0.1
  truststore==0.8.0
  webencodings==0.5.1 (Unable to locate actual module version, using vendor.txt specified version)
Compatible tags: 42
  cp312-cp312-linux_x86_64
  cp312-abi3-linux_x86_64
  cp312-none-linux_x86_64
  cp311-abi3-linux_x86_64
  cp310-abi3-linux_x86_64
  cp39-abi3-linux_x86_64
  cp38-abi3-linux_x86_64
  cp37-abi3-linux_x86_64
  cp36-abi3-linux_x86_64
  cp35-abi3-linux_x86_64
  ...
  [First 10 tags shown. Pass --verbose to show all.]

What's wrong. How do I fix it?

[mustafa0x]

works fine here (mise latest, python 3.12.3, ubuntu 24)

[egnor]

This happens to me as well (also: mise latest, python 3.12.3, ubuntu 24.04)

% mise use -g [email protected]
mise installing precompiled python from indygreg/python-build-standalone
mise if you experience issues with this python, switch to python-build
mise by running: mise settings set python_compile 1
mise [email protected] ✓ installed                                                  mise ~/.config/mise/config.toml tools: [email protected]
mise [email protected] [email protected]                                                 
% which python
/home/egnor/.local/share/mise/installs/python/3.12/bin/python
% which pip
/home/egnor/.local/share/mise/installs/python/3.12/bin/pip
% pip list
Package    Version
---------- -------
pip        24.0
setuptools 69.1.0
% python -m venv my_venv
Error: Command '['/home/egnor/my_venv/bin/python', '-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

digging deeper:

% /home/egnor/my_venv/bin/python -m ensurepip --upgrade --default-pip 
ERROR: Exception:
Traceback (most recent call last):
  File "/tmp/tmpuvohtfdz/pip-24.0-py3-none-any.whl/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/tmp/tmpuvohtfdz/pip-24.0-py3-none-any.whl/pip/_internal/cli/req_command.py", line 245, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmpuvohtfdz/pip-24.0-py3-none-any.whl/pip/_internal/commands/install.py", line 324, in run
    session = self.get_default_session(options)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmpuvohtfdz/pip-24.0-py3-none-any.whl/pip/_internal/cli/req_command.py", line 95, in get_default_session
    self._session = self.enter_context(self._build_session(options))
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmpuvohtfdz/pip-24.0-py3-none-any.whl/pip/_internal/cli/req_command.py", line 122, in _build_session
    session = PipSession(
              ^^^^^^^^^^^
  File "/tmp/tmpuvohtfdz/pip-24.0-py3-none-any.whl/pip/_internal/network/session.py", line 342, in __init__
    self.headers["User-Agent"] = user_agent()
                                 ^^^^^^^^^^^^
  File "/tmp/tmpuvohtfdz/pip-24.0-py3-none-any.whl/pip/_internal/network/session.py", line 150, in user_agent
    zip(["lib", "version"], libc_ver()),
                            ^^^^^^^^^^
  File "/tmp/tmpuvohtfdz/pip-24.0-py3-none-any.whl/pip/_internal/utils/glibc.py", line 84, in libc_ver
    glibc_version = glibc_version_string()
                    ^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmpuvohtfdz/pip-24.0-py3-none-any.whl/pip/_internal/utils/glibc.py", line 8, in glibc_version_string
    return glibc_version_string_confstr() or glibc_version_string_ctypes()
                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmpuvohtfdz/pip-24.0-py3-none-any.whl/pip/_internal/utils/glibc.py", line 43, in glibc_version_string_ctypes
    process_namespace = ctypes.CDLL(None)
                        ^^^^^^^^^^^^^^^^^
  File "/home/egnor/.local/share/mise/installs/python/3.12/lib/python3.12/ctypes/__init__.py", line 379, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: Dynamic loading not supported
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/egnor/.local/share/mise/installs/python/3.12/lib/python3.12/ensurepip/__main__.py", line 5, in <module>
    sys.exit(ensurepip._main())
             ^^^^^^^^^^^^^^^^^
  File "/home/egnor/.local/share/mise/installs/python/3.12/lib/python3.12/ensurepip/__init__.py", line 284, in _main
    return _bootstrap(
           ^^^^^^^^^^^
  File "/home/egnor/.local/share/mise/installs/python/3.12/lib/python3.12/ensurepip/__init__.py", line 200, in _bootstrap
    return _run_pip([*args, *_PACKAGE_NAMES], additional_paths)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/egnor/.local/share/mise/installs/python/3.12/lib/python3.12/ensurepip/__init__.py", line 101, in _run_pip
    return subprocess.run(cmd, check=True).returncode
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/egnor/.local/share/mise/installs/python/3.12/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/egnor/my_venv/bin/python', '-W', 'ignore::DeprecationWarning', '-c', '\nimport runpy\nimport sys\nsys.path = [\'/tmp/tmpuvohtfdz/pip-24.0-py3-none-any.whl\'] + sys.path\nsys.argv[1:] = [\'install\', \'--no-cache-dir\', \'--no-index\', \'--find-links\', \'/tmp/tmpuvohtfdz\', \'--upgrade\', \'pip\']\nrunpy.run_module("pip", run_name="__main__", alter_sys=True)\n']' returned non-zero exit status 2.

This also happens if I install a (precompiled) 3.11. So I don't know what's going on. But it's a real problem!

[mustafa0x]

I wonder whether #262 is related. I suspect it is, since the issue seems to be: the venv for Poetry can't be installed.

[egnor]

[edited]

For me:

% python3
>>> import pip._internal.utils.glibc
>>> pip._internal.utils.glibc.glibc_version_string()
>>>

ok, not sure if None is expected there, but maybe it is since this isn't a glibc build?, anyway it doesn't crash BUT

Python 3.12.3 (main, Apr 15 2024, 18:29:24) [Clang 14.0.3 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib.resources
>>> whl = importlib.resources.files("ensurepip") / "_bundled" / "pip-24.0-py3-none-any.whl"
>>> import sys
>>> sys.path = [str(whl)] + sys.path
>>> import pip._internal.utils.glibc
>>> pip._internal.utils.glibc.glibc_version_string()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/egnor/.local/share/mise/installs/python/3.12/lib/python3.12/ensurepip/_bundled/pip-24.0-py3-none-any.whl/pip/_internal/utils/glibc.py", line 8, in glibc_version_string
  File "/home/egnor/.local/share/mise/installs/python/3.12/lib/python3.12/ensurepip/_bundled/pip-24.0-py3-none-any.whl/pip/_internal/utils/glibc.py", line 43, in glibc_version_string_ctypes
  File "/home/egnor/.local/share/mise/installs/python/3.12/lib/python3.12/ctypes/__init__.py", line 379, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: Dynamic loading not supported

This is using the bundled pip wheel in the same way the pip installation phase in venv creation seems to... and comes to the same end?

[egnor]

heyyyy I think this old bug is related!

In python-build-standalone there's a patch to the linux cpython build which exists to work around an issue that was never properly fixed where a non-dynamically-linked Python couldn't bootstrap pip:

diff --git a/pip/_internal/utils/glibc.py b/pip/_internal/utils/glibc.py
index 819979d80..4ae91e364 100644
--- a/pip/_internal/utils/glibc.py
+++ b/pip/_internal/utils/glibc.py
@@ -47,7 +47,10 @@ def glibc_version_string_ctypes():
     # manpage says, "If filename is NULL, then the returned handle is for the
     # main program". This way we can let the linker do the work to figure out
     # which libc our process is actually using.
-    process_namespace = ctypes.CDLL(None)
+    try:
+        process_namespace = ctypes.CDLL(None)
+    except OSError:
+        return None
     try:
         gnu_get_libc_version = process_namespace.gnu_get_libc_version
     except AttributeError:

That lives on BUT apparently doesn't apply to the pip-24.0-py3-none-any.whl that's bundled with ensurepip and gets used for venv creation:

% unzip -p ~/.local/share/mise/installs/python/3.12/lib/python3.12/ensurepip/_bundled/pip-24.0-py3-none-any.whl pip/_internal/utils/glibc.py | grep -2 CDLL 
        return None

    # ctypes.CDLL(None) internally calls dlopen(NULL), and as the dlopen
    # manpage says, "If filename is NULL, then the returned handle is for the
    # main program". This way we can let the linker do the work to figure out
    # which libc our process is actually using.
    process_namespace = ctypes.CDLL(None)
    try:
        gnu_get_libc_version = process_namespace.gnu_get_libc_version

Note the lack of any try/except around the ctypes.CDLL(None) call -- the patch is not applied there. So that's how it breaks:

• use standalone python build
• go to create a venv
• ensurepip uses its bundled, unpatched pip whl to install itself
• the bundled, unpatched pip whl tries various ways to get the glibc version
• this isn't a glibc build (it uses MUSL) so things like CS_GNU_LIBC_VERSION aren't set
• so the bundled, unpatched pip whl tries to use ctypes.CDLL(None) without a guard
• crash boom etc.

My question is how does this ever work? Clearly it works for @mustafa0x and for that matter used to work for me until recently. Is there some odd path I and @yozachar both followed that leads to us tripping over this pothole?

[egnor]

On a different computer I have access to, I don't see this problem, and the difference appears to be that the precompiled Python that mise is pulling down is the -linux-gnu dynamically linked version (instead of the -linux-musl version) and thus ctypes.CDLL(None) works as usual (but also isn't even needed because os.confstr("CS_GNU_LIBC_VERSION") returns a value). Looking into how that happens.

[egnor]

OK, I think this bug should be retitled: "venv creation fails on linux-musl builds". The cause is the issue with an unpatched embedded pip wheel as described above. The fix is... not obvious to me.

There is a DIFFERENT question which is: why is mise installing linux-musl precompiled Python builds on some unsuspecting Ubuntu systems (which definitely use glibc)? But, that's a mise issue, not a python-build-standalone issue.

[charliermarsh]

Thanks @egnor, that makes sense to me. Is there anything to do in python-build-standalone then?

[charliermarsh]

I guess the change would be... somehow apply that glibc.py patch even when the pip comes from elsewhere? (It makes sense that a dlopen call would fail in the musl build.)

[egnor]

I guess the change would be... somehow apply that glibc.py patch even when the pip comes from elsewhere? (It makes sense that a dlopen call would fail in the musl build.)

I think upstreaming that change would be the best thing! It's clearly a good robustness thing and afaik the only thing that stops pip from working on statically linked Python

[charliermarsh]

Yeah. It looks like there was some work on this in the context of python-build-standalone: pypa/pip#6543 (comment). I guess we'd need to change it in CPython, pip, and setuptools? I haven't dug deeper than reading that thread.

[charliermarsh]

I can try to submit a PR for this in pip.

[egnor]

Yeah I'm not sure the full situation between pip, setuptools, ctypes (in cpython), etc, and it may have evolved since that thread. Certainly that specific patch seems... eminently good?

[egnor]

And, there's no rush since most of us don't need to use the -musl version, and those of us who do don't usually need pip.

[charliermarsh]

Yeah, in python-build-standalone at least I only see that patch on pip, but I'll do some research. Thanks for all your help here.

[charliermarsh]

Ok, it looks like @indygreg actually did patch it in setuptools: pypa/packaging#294

[charliermarsh]

Added in pypa/pip#12716.

[charliermarsh]

Ok, I added this in pip and it has now merged upstream; and we have a patch in python-build-standalone itself, so that's the best we can do IIUC.