regression: --no-build-isolation validation causes failure with setup deps installed from custom commit

[mara004]

Description

After updating from pip 22.0 to 22.1, --no-build-isolation is incompatible with packages installed from a custom commit.
We have setup code that depends on ctypesgen, which did not get a PyPI update for 3 years, while git main is improved regularly and considered stable. Therefore, we install ctypesgen from the latest sources, and then pass --no-build-isolation so that the newer ctypesgen is used, not a PyPI release downloaded into a venv. Since pip 22.1, this fails with the following error:

ERROR: Some build dependencies for file:///home/mara/projects/pypdfium2 conflict with the backend dependencies: ctypesgen==1.0.3.dev98+g2120dbf is incompatible with ctypesgen.

This makes it impossible to install a package via pip3 install while using an unreleased version of a setup dependency.
I am aware that dependency validation was added on purpose, but this particular behaviour is neither justified nor desirable.

Expected behavior

ctypesgen==1.0.3.dev98+g2120dbf fulfils the requirement ctypesgen defined in pyproject.toml:build-system:requires. It should thus not cause an error.

Commit in question: 0c6d20f

pip version

22.1

Python version

3.8.10

OS

Ubuntu 20.04 (Linux)

How to Reproduce

git clone "https://github.com/ctypesgen/ctypesgen.git"
cd ctypesgen
pip3 install .
cd ..
python3 -m pip install pip==22.1
git clone "https://github.com/pypdfium2-team/pypdfium2.git"
cd pypdfium2
pip3 install . -v --no-build-isolation
cd ..
rm -rf ctypesgen pypdfium2

Output

mara@ub2020:/tmp$ git clone "https://github.com/ctypesgen/ctypesgen.git"
Klone nach 'ctypesgen' …
remote: Enumerating objects: 2014, done.
remote: Counting objects: 100% (199/199), done.
remote: Compressing objects: 100% (54/54), done.
remote: Total 2014 (delta 157), reused 145 (delta 145), pack-reused 1815
Empfange Objekte: 100% (2014/2014), 1.11 MiB | 6.06 MiB/s, fertig.
Löse Unterschiede auf: 100% (1295/1295), fertig.
mara@ub2020:/tmp$ cd ctypesgen
mara@ub2020:/tmp/ctypesgen$ pip3 install .
Defaulting to user installation because normal site-packages is not writeable
Processing /tmp/ctypesgen
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: ctypesgen
  Building wheel for ctypesgen (pyproject.toml) ... done
  Created wheel for ctypesgen: filename=ctypesgen-1.0.3.dev98+g2120dbf-py3-none-any.whl size=124318 sha256=b456fcde59ffd9a4b32308b9f9edd79d44dba1a0951b2423b28a66a7b6c8d15f
  Stored in directory: /tmp/pip-ephem-wheel-cache-bahnw45i/wheels/ea/0e/da/227e0ec876339d2ee5e1ec3bf0476fd4eb2076109ca8d08ee0
Successfully built ctypesgen
Installing collected packages: ctypesgen
  Attempting uninstall: ctypesgen
    Found existing installation: ctypesgen 1.0.3.dev98+g2120dbf
    Uninstalling ctypesgen-1.0.3.dev98+g2120dbf:
      Successfully uninstalled ctypesgen-1.0.3.dev98+g2120dbf
Successfully installed ctypesgen-1.0.3.dev98+g2120dbf
mara@ub2020:/tmp/ctypesgen$ cd ..
mara@ub2020:/tmp$ python3 -m pip install pip==22.1
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: pip==22.1 in /home/mara/.local/lib/python3.8/site-packages (22.1)
mara@ub2020:/tmp$ git clone "https://github.com/pypdfium2-team/pypdfium2.git"
Klone nach 'pypdfium2' …
remote: Enumerating objects: 5138, done.
remote: Counting objects: 100% (2084/2084), done.
remote: Compressing objects: 100% (589/589), done.
remote: Total 5138 (delta 1542), reused 1824 (delta 1390), pack-reused 3054
Empfange Objekte: 100% (5138/5138), 1.19 MiB | 2.95 MiB/s, fertig.
Löse Unterschiede auf: 100% (3415/3415), fertig.
mara@ub2020:/tmp$ cd pypdfium2
mara@ub2020:/tmp/pypdfium2$ pip3 install . -v --no-build-isolation
Using pip 22.1 from /home/mara/.local/lib/python3.8/site-packages/pip (python 3.8)
Defaulting to user installation because normal site-packages is not writeable
Processing /tmp/pypdfium2
ERROR: Some build dependencies for file:///tmp/pypdfium2 conflict with the backend dependencies: ctypesgen==1.0.3.dev98+g2120dbf is incompatible with ctypesgen.
mara@ub2020:/tmp/pypdfium2$ cd ..
mara@ub2020:/tmp$ rm -rf ctypesgen pypdfium2
mara@ub2020:/tmp$

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[q0w]

duplicate #11116

[mara004]

duplicate #11116

This is another case broken by the change, but it's different, as you can deduce from the error message. I don't think this issue is a duplicate of it. I like the general idea behind validating dependencies with --no-build-isolation, but this is just a situation where it misbehaves, which should be fixed. Making validation optional is good, but won't fix the actual problem I reported here.

[q0w]

Its the same issue with another case: conflicting reqs. The fix is already merged #11117

[mara004]

This case is claimed to be a conflicting requirement, but it's erroneous:
We have the requirement ctypesgen, and ctypesgen==1.0.3.dev98+g2120dbf is supposed to fulfil this, but pip complains. So while #11117 will restore the previous behaviour, it will not fix the actual bug in the validation.

[q0w]

Srry, yes, its another issue.

[layday]

This is because 1.0.3.dev98+g2120dbf is a pre-release which the packaging library, used to check whether a version exists in a range, discards by default. This:

pip/src/pip/_internal/build_env.py

Line 190 in 30af807

if dist.version not in req.specifier:

... should be:

                if not req.specifier.contains(dist.version, prereleases=True):

[mara004]

Thanks!