Cannot create virtualenv on python3.5 when installed with old pip versions

[srstrickland]

Issue

virtualenv (or at least, a fresh install thereof) is broken for python3.5 due to the combination of:

• a loose version requirement for importlib-resources, which will simply pull the latest available
• the recent publishing of importlib-resources==3.3.0, specifically because of this change
• using an old pip which doesn't find all dependency directives (base ubuntu:16.04 will install pip 8.1.1, for example).

I've verified that pinning importlib-resources<3.3.0 is a viable workaround for this issue. This could be incorporated into virtualenv's setup, or done manually by users wherever necessary. Python 3.5 is end-of-life, and Ubuntu 16.04's end-of-life is fast approaching, so maybe this is not an issue worth fixing. But with such an easy workaround, and since other folks may run into the same, thought I would mention it here.

I only stumbled across this because an open source lib we maintain uses docker + tox testing, and as it hasn't seen much dev over the years, so it still uses ubuntu 16.04 for a base image. I was surprised to see that the same environment setup is no longer functional. By pinning the version as mentioned above, things are functional again.

UPDATE: a better fix is to upgrade pip, which will detect that importlib-resources-3.3.0 does not support py35, via:

python3 -m pip install --upgrade pip

Environment

• OS: Ubuntu 16.04.7 LTS

$ pip list

Package             Version
------------------- -------
appdirs             1.4.4
distlib             0.3.1
filelock            3.0.12
importlib-metadata  2.0.0
importlib-resources 3.3.0
pip                 20.2.4
pytest              3.0.3
pytest-cov          2.3.1
setuptools          20.7.0
six                 1.15.0
sparkly             3.0.0
tox                 2.4.1
virtualenv          20.1.0
wheel               0.29.0
zipp                3.4.0

Output of the virtual environment creation

$ python3.5 -m virtualenv --python /usr/bin/python3.5 myenv -vvv --with-traceback
64 setup logging to NOTSET [DEBUG report:43]
74 find interpreter for spec PythonSpec(path=/usr/bin/python3.5) [INFO builtin:51]
74 proposed PythonInfo(spec=CPython3.5.2.final.0-64, exe=/usr/bin/python3.5, platform=linux, version='3.5.2 (default, Oct  7 2020, 17:19:02) \n[GCC 5.4.0 20160609]', encoding_fs_io=ascii-ANSI_X3.4-1968) [INFO builtin:57]
75 accepted PythonInfo(spec=CPython3.5.2.final.0-64, exe=/usr/bin/python3.5, platform=linux, version='3.5.2 (default, Oct  7 2020, 17:19:02) \n[GCC 5.4.0 20160609]', encoding_fs_io=ascii-ANSI_X3.4-1968) [DEBUG builtin:59]
76 filesystem is case-sensitive [DEBUG info:28]
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.5/dist-packages/virtualenv/__main__.py", line 77, in <module>
    run_with_catch()  # pragma: no cov
  File "/usr/local/lib/python3.5/dist-packages/virtualenv/__main__.py", line 62, in run_with_catch
    run(args, options)
  File "/usr/local/lib/python3.5/dist-packages/virtualenv/__main__.py", line 16, in run
    session = cli_run(args, options)
  File "/usr/local/lib/python3.5/dist-packages/virtualenv/run/__init__.py", line 26, in cli_run
    of_session = session_via_cli(args, options, setup_logging)
  File "/usr/local/lib/python3.5/dist-packages/virtualenv/run/__init__.py", line 42, in session_via_cli
    parser, elements = build_parser(args, options, setup_logging)
  File "/usr/local/lib/python3.5/dist-packages/virtualenv/run/__init__.py", line 70, in build_parser
    ActivationSelector(interpreter, parser),
  File "/usr/local/lib/python3.5/dist-packages/virtualenv/run/plugin/activators.py", line 13, in __init__
    (k, v) for k, v in self.options("virtualenv.activate").items() if v.supports(interpreter)
  File "/usr/local/lib/python3.5/dist-packages/virtualenv/run/plugin/base.py", line 39, in options
    cls._OPTIONS = cls.entry_points_for(key)
  File "/usr/local/lib/python3.5/dist-packages/virtualenv/run/plugin/base.py", line 18, in entry_points_for
    return OrderedDict((e.name, e.load()) for e in cls.entry_points().get(key, {}))
  File "/usr/local/lib/python3.5/dist-packages/virtualenv/run/plugin/base.py", line 18, in <genexpr>
    return OrderedDict((e.name, e.load()) for e in cls.entry_points().get(key, {}))
  File "/usr/local/lib/python3.5/dist-packages/importlib_metadata/__init__.py", line 105, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.5/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 986, in _gcd_import
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 944, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 986, in _gcd_import
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 665, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/usr/local/lib/python3.5/dist-packages/virtualenv/activation/__init__.py", line 3, in <module>
    from .bash import BashActivator
  File "/usr/local/lib/python3.5/dist-packages/virtualenv/activation/bash/__init__.py", line 5, in <module>
    from ..via_template import ViaTemplateActivator
  File "/usr/local/lib/python3.5/dist-packages/virtualenv/activation/via_template.py", line 16, in <module>
    from importlib_resources import read_binary
  File "/usr/local/lib/python3.5/dist-packages/importlib_resources/__init__.py", line 31, in <module>
    from importlib_resources._py3 import (
  File "/usr/local/lib/python3.5/dist-packages/importlib_resources/_py3.py", line 20, in <module>
    Resource = Union[str, os.PathLike]
AttributeError: module 'os' has no attribute 'PathLike'

[pradyunsg]

Given that importlib-resources==3.3.0 correctly declares Requires-Python (https://pypi.org/pypi/importlib-resources/3.3.0/json), I'm a bit surprised that your Python 3.5 environment has importlib-resources 3.3.0.

None the less, pip 20.3 will fix this issue (by fixing pypa/pip#988) which would correctly detect "oh, I made a bad choice" and prevent populating the environment incorrectly.

[gaborbernat]

Not sure what we can do here. Seems this is a bug in your installer, not in our own logic. How did you install virtualenv (exactly)?

[srstrickland]

Good points about pip. My problem can also be summarized as a pip problem; or rather, using an old pip. One which, it seems, does not properly recognize all dependency directives ("don't use 3.3.0 for py35"). This can be replicated easily with the base ubuntu 16.04 image:

docker run -it ubuntu:16.04 /bin/bash

Once inside:

root@2a9bde9bda45:/# apt-get update && apt-get install -y python python-dev python3-pip
<snip>

root@2a9bde9bda45:/# python3 -m pip --version
pip 8.1.1 from /usr/lib/python3/dist-packages (python 3.5)
  --> this is very old, but it's what we get from above.

root@2a9bde9bda45:/# python3 -m pip install virtualenv
<snip>

root@2a9bde9bda45:/# python3 -m pip list
appdirs (1.4.4)
distlib (0.3.1)
filelock (3.0.12)
importlib-metadata (2.0.0)
importlib-resources (3.3.0)
pip (8.1.1)
setuptools (20.7.0)
six (1.15.0)
virtualenv (20.1.0)
wheel (0.29.0)
zipp (3.4.0)
You are using pip version 8.1.1, however version 20.2.4 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

You will notice above that 3.3.0 is installed, and causes virtualenv to break upon use.

By upgrading pip prior to any virtualenv installation, the issue is resolved:

python3 -m pip install --upgrade pip

(note that for my case, I will specifically add ==20.2.4, because I'm paranoid about blind upgrades, and also accept that this may have been resolved anywhere between 8.1.1 and 20.2.4)

There are several things interacting here, but the bottom line from my perspective is this: A set of bootstrapping instructions, which in my mind ought to produce identical (or at least functional) images, became invalid today because of a combination of things. Perhaps it's best practice to always upgrade pip prior to use. While I am hesitant to have all docker images upgrade to the latest pip as part of initial setup, I also understand that asking the python dev community to support all ancient forms of package-installers forever is a silly notion.

I'm happy that there is a better fix than pinning versions manually, and mostly wanted to share in case people ran into the same.

I will update the title to reflect these learnings; please feel free to close the issue.