pip 20.3.2 eagerly downloads all versions of a package during dependency resolution #9284
Comments
|
Probably related: |
|
@pradyunsg, @uranusjr, and @NoahGorny this is a really bad bug 😬 |
|
I just ran into this myself when trying to upgrade my version of pandas (1.1.3) to the latest version (currently 1.1.5). And then saw that it did the same thing when trying to upgrade pip itself. Out of morbid curiosity, I let the pandas upgrade run its course inside a fresh virtual environment. It took over 35 minutes and downloaded over 650 MB of old packages into the cache directory: https://pastebin.com/V6kg6tTX Edit: Moved code to as per @omry. |
|
It should be noted, it's not an infinite loop... Here's the full log of pip 20.3.2 trying to upgrade itself (adds 157 MB of old versions into the cache): https://pastebin.com/AFny1enP Edit: Moved code to as per @omry. |
|
can you use pastebin when pasting very long pastes? it makes looking for comments here very painful. |
|
Oh my god, |
|
Gosh...Plz...CI System bombed by pip... |
|
This is the mitigation I'm using until the bug is fixed. |
|
Exactly |
|
|
|
I did some bisecting, and the issue is indeed from a resolver change, specifically in d45541c. The issue was not identified in tests because it requires a set of conditions to trigger (I have not figured out the exact conditions). |
ValueError: Unable to find resource t64.exe in package pip._vendor.distlib |
|
Oh.. I should have seen this issue before I struggle like this ... |
|
@tiantian1645 This is a common issue when you try to roll back a broken pip version. See #8450 (comment) for steps to rescure. |
This comment has been minimized.
This comment has been minimized.
|
People affected by this: Could you try and see if it behaves correctly? |
|
seem to be good... but with version 21.0.dev0? I can not check if this fix "no ignore yanked" bug... can you make this package with 20.3.1 number of version to check that? |
|
You can temporary workaround it by using the old resolver: |
|
|
|
Since this is not an infinite loop, please change the title of this issue to prevent further duplicates. |
pradyunsg
changed the title
|
Oh boi. This is "fun". Sorry for the disruption folks! Let's revert d45541c, and then take things from there? |
|
for clarity, if you are currently affected by this, and your pip is barfing endlessly, run whichever corresponds to your permissions and pip version: |
|
pip 20.3.3 is still dumping 23 Gb of dependencies: Example: This is not scalable. Before things worked fine without downloading so many packages. |
|
pip 20.3.3 works fine for me on macOS 11.1. @pabloa I ran |
|
Yes, the issue is triggered by the I’ll close this since the eager download issue seems fixed. The “pip downloads many versions” issue is tracked in #9187. |
|
I tested with an existent and a new virtual environment. The last report is from a CI server that deletes everything and creates a new virtual environment. |
|
Yeah, seems to be fixed. Going from 20.3.1 to 20.3.3 with |
|
Thanks for the fix on this... I was starting to think I was crazy. And for the search engines: pip downloading lots of old versions |
|
Does anyone know how to install 20.3.3 if you have 20.3.2? I try to upgrade and it keeps pulling every version. |
|
Nevermind, |
|
FYI (and to capture the experience), I also got the slightly deranged experience when attempting to upgrade from 20.3.2: Good suggestion to use |
|
this still happens on the latest version, thats crazy |
|
What's the output of |
|
Try getting the latest version by running |
|
i tried 21.0.1 and 20.3.3, no change, the same behaviour |
|
(venv) oli@lucy:~/Downloads/virtual_webcam_background-master$ pip -V |
|
Yep, I confirm it's back. It's happening to me as well as @damnms: @pradyunsg @uranusjr I though we had this fixed? |
|
The key word of this issue is “eagerly”. At the time, pip would download all versions of a package, only to find that it does not need it after all. That was fixed. In your scenario, pip downloads multiple versions because the versions it downloaded conflict with other packages you specified, which is called backtracking and is a feature. The versions need to be downloaded to detect the conflicts. You are misidentifying the issue. |
|
@uranusjr I think what would greatly help to identify such cases is if pip explained why it is downloading (and discarding) multiple versions then. Even in the most verbose mode, I cannot find any such information, and it's filling hundreds of terminal pages. Note that I cannot current repro this problem with to tell me why Couldn't it say something like
Here's my output of |
|
tbh i just want to run this piece of software on debian 11: anyone an idea how to get this working when pip behaves like it does? should i create a new report to not spam this one here? unfortunately we use a lot of conferencing software (zoom, ms teams, etc.) and i dont want anyone be able to see my room etc. |
|
@damnms Probably the issue tracker for the software itself. |
|
Is pip broken again? a I submitted a report to https://pip.pypa.io/surveys/backtracking but that doesn't really help me right now. |
|
It is not broken again. Please read this: #9284 (comment) I’m going to lock this thread to prevent people from asking again. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
and others
Environment
Description
If you use pip to update requirements after installing 20.3.2 it goes into an infinite loop resolving and downloading previous versions of dependencies.
This happened on our CI environment that took 26 minutes to complete a job that takes no longer than 3, but then we were able to replicate it locally by just telling pip to update himself...
Expected behavior
Dependencies should be resolved without seeking all versions of a package.
How to Reproduce
pip install -U pipto get to 20.3.2pip install -U pipagainOutput
First, update pip
Then update it again (I had to ctrl+C to stop the loop)
The text was updated successfully, but these errors were encountered: