pip sometimes prohibits 2 .dist-info directories in wheels

[chrahunt]

Environment

• pip version: 19.3.1
• Python version: 3.8.0
• OS: Linux

Description

Currently, pip rejects wheel files containing multiple .dist-info directories, but only if they start with the name of the package being installed.

Expected behavior

pip should prohibit more than 1 top-level .dist-info directory, without regard to the name of the directories (besides ending with .dist-info).

From PEP 427:

A wheel is a ZIP-format archive with a specially formatted file name and the .whl extension. It contains a single distribution

And from PEP 376:

One .dist-info directory per installed distribution

How to Reproduce

Save t.sh below, make it executable, and execute it.

t.sh

#!/bin/sh
cd "$(mktemp -d)"
python -m venv env
. env/bin/activate
pip install --upgrade pip wheel

echo "= Versions ====================================================================="
python -V
pip -V
wheel version

echo "= Setup ========================================================================"
echo "from setuptools import setup; setup(name='hello')" > setup.py

echo "= (1) Base case ================================================================"
echo "= (1) Build ===================================================================="
python setup.py bdist_wheel

echo "= (1) Install =================================================================="
pip install --ignore-installed dist/hello-0.0.0-py3-none-any.whl

echo "= (2) 2 .dist-info dirs (expected) ============================================="
echo "= (2) Build ===================================================================="
python setup.py bdist_wheel

echo "= (2) Add second .dist-info (shared prefix) ===================================="
mkdir hello-example-0.0.0.dist-info
zip dist/hello-0.0.0-py3-none-any.whl hello-example-0.0.0.dist-info

echo "= (2) Install =================================================================="
pip install --ignore-installed dist/hello-0.0.0-py3-none-any.whl

echo "= (3) 2 .dist-info dirs (NOT expected) ========================================="
echo "= (3) Build ===================================================================="
python setup.py bdist_wheel

echo "= (3) Add second .dist-info (no shared prefix) ================================="
mkdir goodbye-example-0.0.0.dist-info
zip dist/hello-0.0.0-py3-none-any.whl goodbye-example-0.0.0.dist-info

echo "= (3) Install =================================================================="
pip install --ignore-installed dist/hello-0.0.0-py3-none-any.whl

Output

Output

Collecting pip
  Using cached https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl
Collecting wheel
  Using cached https://files.pythonhosted.org/packages/00/83/b4a77d044e78ad1a45610eb88f745be2fd2c6d658f9798a15e384b7d57c9/wheel-0.33.6-py2.py3-none-any.whl
Installing collected packages: pip, wheel
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-19.3.1 wheel-0.33.6
= Versions =====================================================================
Python 3.8.0
pip 19.3.1 from /tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip (python 3.8)
wheel 0.33.6
= Setup ========================================================================
= (1) Base case ================================================================
= (1) Build ====================================================================
running bdist_wheel
running build
installing to build/bdist.linux-x86_64/wheel
running install
running install_egg_info
running egg_info
creating hello.egg-info
writing hello.egg-info/PKG-INFO
writing dependency_links to hello.egg-info/dependency_links.txt
writing top-level names to hello.egg-info/top_level.txt
writing manifest file 'hello.egg-info/SOURCES.txt'
reading manifest file 'hello.egg-info/SOURCES.txt'
writing manifest file 'hello.egg-info/SOURCES.txt'
Copying hello.egg-info to build/bdist.linux-x86_64/wheel/hello-0.0.0-py3.8.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/hello-0.0.0.dist-info/WHEEL
creating 'dist/hello-0.0.0-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'hello-0.0.0.dist-info/METADATA'
adding 'hello-0.0.0.dist-info/WHEEL'
adding 'hello-0.0.0.dist-info/top_level.txt'
adding 'hello-0.0.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
= (1) Install ==================================================================
Processing ./dist/hello-0.0.0-py3-none-any.whl
Installing collected packages: hello
Successfully installed hello-0.0.0
= (2) 2 .dist-info dirs (expected) =============================================
= (2) Build ====================================================================
running bdist_wheel
running build
installing to build/bdist.linux-x86_64/wheel
running install
running install_egg_info
running egg_info
writing hello.egg-info/PKG-INFO
writing dependency_links to hello.egg-info/dependency_links.txt
writing top-level names to hello.egg-info/top_level.txt
reading manifest file 'hello.egg-info/SOURCES.txt'
writing manifest file 'hello.egg-info/SOURCES.txt'
Copying hello.egg-info to build/bdist.linux-x86_64/wheel/hello-0.0.0-py3.8.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/hello-0.0.0.dist-info/WHEEL
creating 'dist/hello-0.0.0-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'hello-0.0.0.dist-info/METADATA'
adding 'hello-0.0.0.dist-info/WHEEL'
adding 'hello-0.0.0.dist-info/top_level.txt'
adding 'hello-0.0.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
= (2) Add second .dist-info (shared prefix) ====================================
  adding: hello-example-0.0.0.dist-info/ (stored 0%)
= (2) Install ==================================================================
Processing ./dist/hello-0.0.0-py3-none-any.whl
Installing collected packages: hello
ERROR: Exception:
Traceback (most recent call last):
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 153, in _main
    status = self.run(options, args)
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 446, in run
    installed = install_given_reqs(
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/req/__init__.py", line 58, in install_given_reqs
    requirement.install(
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 858, in install
    self.move_wheel_files(
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 487, in move_wheel_files
    wheel.move_wheel_files(
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/wheel.py", line 461, in move_wheel_files
    clobber(source, lib_dir, True)
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/wheel.py", line 408, in clobber
    assert not info_dir, ('Multiple .dist-info directories: ' +
AssertionError: Multiple .dist-info directories: /tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/hello-example-0.0.0.dist-info, /tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/hello-0.0.0.dist-info
= (3) 2 .dist-info dirs (NOT expected) =========================================
= (3) Build ====================================================================
running bdist_wheel
running build
installing to build/bdist.linux-x86_64/wheel
running install
running install_egg_info
running egg_info
writing hello.egg-info/PKG-INFO
writing dependency_links to hello.egg-info/dependency_links.txt
writing top-level names to hello.egg-info/top_level.txt
reading manifest file 'hello.egg-info/SOURCES.txt'
writing manifest file 'hello.egg-info/SOURCES.txt'
Copying hello.egg-info to build/bdist.linux-x86_64/wheel/hello-0.0.0-py3.8.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/hello-0.0.0.dist-info/WHEEL
creating 'dist/hello-0.0.0-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'hello-0.0.0.dist-info/METADATA'
adding 'hello-0.0.0.dist-info/WHEEL'
adding 'hello-0.0.0.dist-info/top_level.txt'
adding 'hello-0.0.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
= (3) Add second .dist-info (no shared prefix) =================================
  adding: goodbye-example-0.0.0.dist-info/ (stored 0%)
= (3) Install ==================================================================
Processing ./dist/hello-0.0.0-py3-none-any.whl
Installing collected packages: hello
Successfully installed hello-0.0.0

The unexpected case is (3) Install - the installation should not go through since there are multiple .dist-info directories.

[uranusjr]

Does pip allow multiple dist-info dirs (in general) to handle installing from flat directory? (It’s the case IIRC.) I kind of feel the responsibility to produce a valid wheel should fall on to the wheel builder (so build backend).

pip can be free to interpret and invalid wheel and do what’s most comfortable (e.g. choose one by whatever logic, as the current behaviour) since the input already breaks the contract. It is also a valid behaviour to error out as well, but I wouldn’t say it’s necessary.

[pradyunsg]

I think erroring out results in the best end-user experience here -- you gave pip something invalid, so it did the wrong thing vs you gave pip something invalid and it refused to use it.

[chrahunt]

Does pip allow multiple dist-info dirs (in general) to handle installing from flat directory?

Good question. The current behavior is here - we essentially guess. I think this use case is different from installing a wheel because the wheel format has a spec and we control everything about the directory it is unpacked into.

I kind of feel the responsibility to produce a valid wheel should fall on to the wheel builder (so build backend).

Agreed.

pip can be free to interpret and invalid wheel and do what’s most comfortable

I am trying to refactor some of our wheel installation logic. The purpose of calling out the expected behavior here is to make it easier to make things more comfortable. :)

Our current validation approach has lead to situations like this one, but extracting it while preserving the exact same behavior would result in code that is harder to follow and doesn't really map to the spec. I expect taking the stricter approach will lead to fewer "print better errors" issues, since we can print a nice error upfront, at least in this case.

[uranusjr]

I’m totally on board if the behaviour change would improve the code quality.

[chrahunt]

I checked all wheels associated with the most recent release of all projects on PyPI and it looks like only intel-tensorflow would be affected by this change. I couldn't find any good contact information for that project after a few minutes of looking, since it just copies the existing tensorflow metadata. Maybe https://github.com/Intel-tensorflow/tensorflow is the repo?

In any case a possible backwards-compatible workaround:

1. upload old wheels with a new build number if needed
2. put non-primary .dist-info into {distribution}-{version}.data/purelib/