Improve uninstall logic to handle file removal failure e.g. pip install -U pip on Windows

[uranusjr]

When pip is uninstalling (or upgrading/reinstalling) itself, it would try to remove the wrapper script generated by the installation. This causes problems on Windows when you run something like pip upgrade pip because it is an error to delete the executable of a running process.

Currently pip tries to avoid this by detecting how the user invokes pip, and error-ing out early before the uninstallation happens on certain situations (pip._internal.utils.misc.protect_pip_from_modification_on_windows). But the checks themselves have many edge cases, and we’ve been either not catching things (#6841) or overly strict (#7358). I believe it is possible to implement a comprehensive check (#7558), but it would be quite complex. @chrahunt suggested in #7558 (comment) that it would be a good idea to get rid of these special cases, and fix the removal failures instead.

Since pip already moves them to a temporary directory during uninstallation (so it can rollback on e.g. upgrade failures), the missing piece here is to not error on when the stashed files fail to be deleted. This is done by TempDirectory.cleanup, which simply calls rmtree on the temp dir, so one approach to solve this would be to add an onerror handler that moves the failed-to-be-deleted file somewhere (or just keep it and print out its location).

[pradyunsg]

/cc @pfmoore

[pfmoore]

This certainly seems like a viable idea. A quick test by putting a try...except pass around the rmtree here does cause pip install -U to work cleanly. Obviously a real fix should be more careful (maybe even to the extent of special-casing pip*.exe, as they are the only files where we expect deletion to fail?)

I don't think I had ever realised we moved the files before upgrading. Good catch!

I'm slightly sad that we'll routinely leave clutter behind in $TEMP, but not sad enough to say we shouldn't do this 🙂

[uranusjr]

as they are the only files where we expect deletion to fail?

I hit this often by accident, e.g. upgrading flake8 when it is in use by VS Code. I accept them as user errors, but it’d help improve my workflow a lot if we do this for all files.

[pfmoore]

Fair enough, I don't have a problem with that. Maybe trap "Access denied" and filename is *.exe to just hit "trying to delete an in-use Windows exe wrapper"? But honestly, "don't get upset if cleanup fails" is just as good a principle, so I'm not going to argue here.

[uranusjr]

I decided to implement something for this since this came up again and I don’t have better things to do now 🙂

[SigmundVik]

I ran into the same issue when switching between different versions of pip. With the knowledge from this thread the workaround is obvious, just use python.exe to invoke pip:

[BOOTCAMP] [none] Thu 11-11-21 16:06 8.600 23G z:\>pip --version
pip 21.3 from C:\tools\Python39\lib\site-packages\pip (python 3.9)

[BOOTCAMP] [none] Thu 11-11-21 16:07 0.640 23G z:\>pip install pip==21.2.4
Collecting pip==21.2.4
  Using cached pip-21.2.4-py3-none-any.whl (1.6 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.3
    Uninstalling pip-21.3:
      Successfully uninstalled pip-21.3
ERROR: Could not install packages due to an OSError: [WinError 5] Access is denied: 'C:\\tools\\Python39\\~3ripts\\pip.exe'
Consider using the `--user` option or check the permissions.


[BOOTCAMP] [none] Thu 11-11-21 16:07 8.632 23G z:\>pip --version
pip 21.2.4 from C:\tools\Python39\lib\site-packages\pip (python 3.9)

[BOOTCAMP] [none] Thu 11-11-21 16:07 0.634 23G z:\>python -m pip install pip==21.3
Collecting pip==21.3
  Using cached pip-21.3-py3-none-any.whl (1.7 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.2.4
    Uninstalling pip-21.2.4:
      Successfully uninstalled pip-21.2.4
Successfully installed pip-21.3

[pradyunsg]

This has been fixed in #10560 and will be in the next release. Users will benefit from the fix in releases after that.

Users hitting this should run python -m pip install --upgrade pip --force-reinstall to fix their Python installation.

If you're unable to do so for some reason, download get-pip.py and run it with --force-reinstall.

[pradyunsg]

I'd suggest reporting that to the brotli package.

[pradyunsg]

@89z let's discuss that in #10950. :)