Remove the setuptools-provided distutils hack, if using distutils

[pradyunsg]

This ensures that pip's imported copy of distutils comes from the
standard library, if/when the hack needs to be used.

Closes #11294
Closes #8761

[pradyunsg]

To test this manually, you can pip install https://github.com/pradyunsg/pip/archive/refs/heads/remove-distutils-shim.zip

[jaraco]

In #8761 (comment), I tested the change and it's having the desired effect. Thanks!

[Jorricks]

As mentioned here, this fixes the issue I am currently fixing with Flit.

[uranusjr]

I wonder if we should instead back-port the distutils removal PRs to 22.2. We are already committed to doing it, and it feels somewhat unnecessary to maintain a different fix to the same issue in 22.2.

[sbidoul]

I wonder if we should instead back-port the distutils removal PRs to 22.2. We are already committed to doing it, and it feels somewhat unnecessary to maintain a different fix to the same issue in 22.2.

@uranusjr I think all of them are in 22.2, unless I missed one that is not merged yet ?

[uranusjr]

Ah right, the problem here is that some parts of pip have to continue using distutils (location-related stuff) and we still need a fix for those. So this PR is still neccesary.

[pradyunsg]

the problem here is that some parts of pip have to continue using distutils (location-related stuff)

Huh?

22.2 has all the distutils removal PRs we created, and distutils won't get imported on 3.10 and above. However, we need to use distuils on 3.7-3.9, because we know that many redistributors haven't configured sysconfig correctly.

This PR is necessary since it's placing the shim-removal logic in the "correct" location within pip, which knows when it's going to import distutils (limited to a single module -- in _locations/_distutils.py) unlike setuptools which needs to do the same thing in a .pth file instead. The setuptools approach doesn't work with the removal of create-a-zipfile approach to making pip importable in an isolated environment, only because it tries to remove the shim as a side-effect of a lookup for a pip import -- a lookup that never reaches the setuptools import hook because pip's redirector handles it before it does.

[sbidoul]

@pradyunsg do you feel like adding a news entry that is understandable by mere humans? :) I personally wouldn't know how to formulate that 🤯

[pradyunsg]

I have made an attempt, at communicating this in the NEWS file. Honestly, I'm not quite sure how we'd test this behaviour within our test suite.

@sbidoul How strongly do you feel about adding a test to the test suite for #11294? :)

[sbidoul]

@sbidoul How strongly do you feel about adding a test to the test suite for #11294? :)

I can live without a test for that :) especially since I would not know either where to start to add one. I'm still perplexed that our current test suite does not reveal it.

[bdrosen96]

This seems to cause issues because it can raise AttributeError if used with a distutils version without remove_shim

14:16:36      __import__("_distutils_hack").remove_shim()
14:16:36  AttributeError: module '_distutils_hack' has no attribute 'remove_shim'

[pradyunsg]

What version of setuptools do you have?

[bdrosen96]

setuptools==36.0.1

[bdrosen96]

This errors occurs when running:

python3 -m pip install -U pip setuptool

[pradyunsg]

That attribute has been in setuptools for over two years now: https://github.com/pypa/setuptools/blame/main/_distutils_hack/__init__.py#L218

It seems weird to have an environment with such an old version of setuptools and the latest pip release that was released mere hours ago.

[pradyunsg]

Could you please file an issue with the details, and possibly a link to a publicly viewable failure log?

[jaraco]

I was tempted to wrap the workaround in a failsafe, but I presumed it was unlikely that any user would have this combination (it's common and recommended for users to always upgrade to the latest Setuptools). Perhaps it deserves a failsafe (try/except), or perhaps it should be unsupported. Sorry for the hassle.