Install namespace package with --prefix does not install the actual package files

[NiklasRosenstein]

• Pip version: 18.0 (also tested in 10.0.1)
• Python version: 3.6.5
• Operating system: Windows 10

$ cd ~/Desktop
$ pip install --prefix test --ignore-installed hurry.filesize
$ ls test/Lib/site-packages/
__pycache__/  easy_install.py  hurry.filesize-0.9-py3.6.egg-info/ pkg_resources/  setuptools/  setuptools-40.0.0.dist-info/
$ cat test/Lib/site-packages/hurry.filesize-0.9-py3.6.egg-info/installed-files.txt
..\hurry.filesize-0.9-py3.6-nspkg.pth
..\hurry\filesize\README.txt
..\hurry\filesize\__init__.py
..\hurry\filesize\__pycache__\__init__.cpython-36.pyc
..\hurry\filesize\__pycache__\filesize.cpython-36.pyc
..\hurry\filesize\__pycache__\tests.cpython-36.pyc
..\hurry\filesize\filesize.py
..\hurry\filesize\tests.py
PKG-INFO
SOURCES.txt
dependency_links.txt
namespace_packages.txt
not-zip-safe
requires.txt
top_level.txt
$ ls test/Lib/site-packages/hurry
ls: cannot access 'test/Lib/site-packages/hurry': No such file or directory

You can see that the hurry directory doesn't actually exist – even though the installed-files.txt lists it with the package files as being installed.

Edit 1 fixed Python version

[NiklasRosenstein]

On a side-note, shouldn't --prefix become a global option so that it can be used with uninstall, freeze, show etc.?

[NiklasRosenstein]

Here's the full output from the installation: https://pastebin.com/7eh0p4Jx (too long for GitHub comments)

[benoit-pierre]

Can you reproduce with:

> rm -rf prefix && ./venv/bin/pip install --no-cache --prefix prefix --no-deps hurry.filesize
> find prefix
prefix
prefix/lib
prefix/lib/python3.7
prefix/lib/python3.7/site-packages
prefix/lib/python3.7/site-packages/hurry.filesize-0.9-py3.7.egg-info
prefix/lib/python3.7/site-packages/hurry.filesize-0.9-py3.7.egg-info/installed-files.txt

It works fine without --no-cache, or when passing an absolute path to --prefix (e.g. --prefix $PWD/prefix).

[NiklasRosenstein]

Yes, I can reproduce it with this command. Any idea where to start looking for the cause?

[chrahunt]

This is easy to reproduce with the latest pip and a non-namespace package.

The problem is that for the legacy code path (what we do for non-PEP 517 packages when wheel is not installed), we run the setup.py from the directory it is contained in, but the prefix we pass to setuptools is still relative.

t.sh

#!/bin/sh

cd "$(mktemp -d)"

python -m venv env
./env/bin/pip install --upgrade pip

./env/bin/python -V
./env/bin/pip -V

cat <<EOF > setup.py
from setuptools import setup

setup(
    name='hello',
    version='0.1.0',
)
EOF

./env/bin/pip install --prefix prefix .
find prefix

Output

Collecting pip
  Using cached https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-19.3.1
Python 3.8.0
pip 19.3.1 from /tmp/user/1000/tmp.geF3LtE9WO/env/lib/python3.8/site-packages/pip (python 3.8)
Processing /tmp/user/1000/tmp.geF3LtE9WO
Installing collected packages: hello
    Running setup.py install for hello ... done
Successfully installed hello-0.1.0
prefix
prefix/lib
prefix/lib/python3.8
prefix/lib/python3.8/site-packages
prefix/lib/python3.8/site-packages/hello-0.1.0-py3.8.egg-info
prefix/lib/python3.8/site-packages/hello-0.1.0-py3.8.egg-info/installed-files.txt

The easiest fix is just to make --prefix absolute. This could be handled in a callback for the --prefix option, similar to what was done in #7467.

We would also need a test for this, which would look like test_install_package_with_prefix. We should assert that an .egg-info directory is created to make sure that we were using the legacy code path.

[Mr-Pepe]

Quick update: This issue will be obsolete when setup.py install gets deprecated, which is scheduled for April 2023 according to #8559.

[sbidoul]

With 23.1 and #11874 around the corner we can close this.