Pip always upgrades VCS packages

[ohadber]

Description

If we'll use pip to install a package which resides in VCS, it will try to upgrade the dependency every time we perform pip install (even if we'll not use the -U flag). Note that it includes cloning the repository every time.

Important note: While using the deprecated resolver this bug doesn't reproduce(I am currently using the deprecated legacy resolver in order to avoid re-cloning every time).

Expected behavior

The expected behavior is that when using pip install without the -U flag i want to avoid upgrading already-existing packages. In addition I wouldn't like to clone them all over again to save time.

pip version

21.3.1

Python version

3.8.10

OS

Ubuntu 20.04

How to Reproduce

1. Use this setup.py

from setuptools import setup

setup(
    name="testing-git",
    install_requires=[
        "requests @ git+ssh://[email protected]/psf/requests.git"
    ]
)

2. go into the setup.py directory.
3. execute pip install .
4. execute pip install . again.

Output

Output for 3:

(venv) ohadberenstein@EU-99SB6J3:~/Dev/tmp$ pip install .
Processing /home/ohadberenstein/Dev/tmp
  Preparing metadata (setup.py) ... done
Collecting requests@ git+ssh://[email protected]/psf/requests.git
  Cloning ssh://****@github.com/psf/requests.git to /tmp/pip-install-6ab39s_e/requests_2e3aa7f4f37e4e3996ab901e0e39e26f
  Running command git clone --filter=blob:none -q 'ssh://****@github.com/psf/requests.git' /tmp/pip-install-6ab39s_e/requests_2e3aa7f4f37e4e3996ab901e0e39e26f
  Resolved ssh://****@github.com/psf/requests.git to commit 24dfbfde552bb8ce48eec415714cbe793cfa95bd
  Preparing metadata (setup.py) ... done
Collecting certifi>=2017.4.17
  Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.8-py2.py3-none-any.whl (138 kB)
Collecting charset_normalizer~=2.0.0
  Using cached charset_normalizer-2.0.10-py3-none-any.whl (39 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.3-py3-none-any.whl (61 kB)
Using legacy 'setup.py install' for testing-git, since package 'wheel' is not installed.
Using legacy 'setup.py install' for requests, since package 'wheel' is not installed.
Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests, testing-git
    Running setup.py install for requests ... done
    Running setup.py install for testing-git ... done
Successfully installed certifi-2021.10.8 charset-normalizer-2.0.10 idna-3.3 requests-2.27.1 testing-git-0.0.0 urllib3-1.26.8```

Output for 4:

(venv) ohadberenstein@EU-99SB6J3:~/Dev/tmp$ pip install .
Processing /home/ohadberenstein/Dev/tmp
  Preparing metadata (setup.py) ... done
Collecting requests@ git+ssh://[email protected]/psf/requests.git
  Cloning ssh://****@github.com/psf/requests.git to /tmp/pip-install-r2yqeb7x/requests_87f963b69ea64e72aab2d593f44da0e7
  Running command git clone --filter=blob:none -q 'ssh://****@github.com/psf/requests.git' /tmp/pip-install-r2yqeb7x/requests_87f963b69ea64e72aab2d593f44da0e7
  Resolved ssh://****@github.com/psf/requests.git to commit 24dfbfde552bb8ce48eec415714cbe793cfa95bd
  Preparing metadata (setup.py) ... done
Requirement already satisfied: certifi>=2017.4.17 in ./venv/lib/python3.8/site-packages (from requests@ git+ssh://[email protected]/psf/requests.git->testing-git==0.0.0) (2021.10.8)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./venv/lib/python3.8/site-packages (from requests@ git+ssh://[email protected]/psf/requests.git->testing-git==0.0.0) (1.26.8)
Requirement already satisfied: charset_normalizer~=2.0.0 in ./venv/lib/python3.8/site-packages (from requests@ git+ssh://[email protected]/psf/requests.git->testing-git==0.0.0) (2.0.10)
Requirement already satisfied: idna<4,>=2.5 in ./venv/lib/python3.8/site-packages (from requests@ git+ssh://[email protected]/psf/requests.git->testing-git==0.0.0) (3.3)
Using legacy 'setup.py install' for testing-git, since package 'wheel' is not installed.
Installing collected packages: testing-git
  Attempting uninstall: testing-git
    Found existing installation: testing-git 0.0.0
    Uninstalling testing-git-0.0.0:
      Successfully uninstalled testing-git-0.0.0
    Running setup.py install for testing-git ... done
Successfully installed testing-git-0.0.0

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[webknjaz]

"requests @ git+ssh://[email protected]/psf/requests.git"

I don't think putting URLs in Python package distribution metadata is considered a good practice. These things should go to requirements files while the metadata should have just package names (and version boundaries).

I suppose that the behavior is like this because you are not pointing at a specific software version but rather at an unspecified "default" branch of a remote repository which will point at different versions (commits) over time. Also, even the branch name that is considered default for a given Git remote may change over time too (master -> main -> develop -> devel -> dev).

There is no way for pip to know if the version is the same or not because the way it's requested points at different versions of the codebase at the same time. I guess it could be optimized a bit by caching the commit hash and checking if the branch pointer changed via git ls-remote https://github.com/psf/requests.git HEAD | head -n1. OTOH, this would still mean that your new teammates will probably get a different version of requests than you (having set up your environment months before).

I would recommend trying to specify a full commit SHA to pin it for reproducibility.

[webknjaz]

Thinking about it more, since VCS packages get built from the source, this means that you may get a different result depending on your environment even if the commit hash does not change. Maybe pip could add an explanatory note to the output mentioning why the rebuild is required.

[ohadber]

I think we are not on the same page. I'll write an example to make sure that we are aligned:

Lets say I have package requests in some version, not the latest one. If i will execute pip install requests it won't be upgraded to the latest version. The reason for that is because i did not specify the -U flag. I expect the same to happen with VCS packages - if i'm installing a package which is already installed, it shouldn't be upgraded unless i specify the -U flag.

(venv) ohadberenstein@EU-99SB6J3:~/tmp/venv/bin$ pip list | grep requests
requests                           2.26.0
requests-toolbelt                  0.9.1
(venv) ohadberenstein@EU-99SB6J3:~/tmp/venv/bin$ pip install requests
Requirement already satisfied: requests in /home/ohadberenstein/Dev/v1/venv/lib/python3.8/site-packages (2.26.0)
Requirement already satisfied: idna<4,>=2.5 in /home/ohadberenstein/Dev/v1/venv/lib/python3.8/site-packages (from requests) (3.3)
Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ohadberenstein/Dev/v1/venv/lib/python3.8/site-packages (from requests) (2.0.9)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ohadberenstein/Dev/v1/venv/lib/python3.8/site-packages (from requests) (1.22)
Requirement already satisfied: certifi>=2017.4.17 in /home/ohadberenstein/Dev/v1/venv/lib/python3.8/site-packages (from requests) (2021.10.8)

please notice that there is a newer requests version exists(requests-2.27.1).

[webknjaz]

How would pip know/verify that the thing that a remote Git repository has contains a distribution called requests? It'd need to build the repository via the PEP 517 interface and then, see what it produces. Only then, it'll be able to check the metadata of the thing, like the dist name and the version. Until this happens, pip has no way of knowing if the repo you're pointing at has a thing you've got installed on disk.

[ohadber]

As far as i understand, the egg name should indicate what's the package name(and i think the prefix before the @ in the dependency list is the egg name. not 100% sure of it though).

pip looks at 2 fragments for VCS URLs:
egg: For specifying the “project name” for use in pip’s dependency resolution logic. eg: egg=project_name

Reference can be found here:
https://pip.pypa.io/en/stable/topics/vcs-support/#url-fragments

For example, in the following setup.py the name before the @ is requests, then pip should know he is installing the requests package.

from setuptools import setup

setup(
    name="testing-git",
    install_requires=[
        "requests @ git+ssh://[email protected]/psf/requests.git"
    ]
)

[pradyunsg]

Duplicate of #5780, in the sense that the discussion there covers this topic.

Please search the issue tracker for duplicates before filing issues.