Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests failing to run with BackendUnavailable since pip 19 #1644

Closed
jaraco opened this issue Jan 25, 2019 · 31 comments · Fixed by #2582
Closed

Tests failing to run with BackendUnavailable since pip 19 #1644

jaraco opened this issue Jan 25, 2019 · 31 comments · Fixed by #2582
Labels

Comments

@jaraco
Copy link
Member

jaraco commented Jan 25, 2019

It seems that since the release of pip 19, Setuptools' own tests fail to run thus:

setuptools fix_889_and_non-ascii_in_setup.cfg_take_2 $ tox -e py27                                                                                                                       
py27 create: /Users/jaraco/code/main/setuptools/.tox/py27
py27 installdeps: -rtests/requirements.txt
py27 develop-inst: /Users/jaraco/code/main/setuptools
ERROR: invocation failed (exit code 2), logfile: /Users/jaraco/code/main/setuptools/.tox/py27/log/py27-2.log
ERROR: actionid: py27
msg: developpkg
cmdargs: '/Users/jaraco/code/main/setuptools/.tox/py27/bin/pip install --exists-action w -e /Users/jaraco/code/main/setuptools'

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Obtaining file:///Users/jaraco/code/main/setuptools
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
Exception:
Traceback (most recent call last):
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 315, in run
    resolver.resolve(requirement_set)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/resolve.py", line 131, in resolve
    self._resolve_one(requirement_set, req)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/resolve.py", line 226, in _get_abstract_dist_for
    req, self.require_hashes, self.use_user_site, self.finder,
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 382, in prepare_editable_requirement
    abstract_dist.prep_for_dist(finder, self.build_isolation)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 149, in prep_for_dist
    reqs = self.req.pep517_backend.get_requires_for_build_wheel()
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_vendor/pep517/wrappers.py", line 71, in get_requires_for_build_wheel
    'config_settings': config_settings
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_vendor/pep517/wrappers.py", line 162, in _call_hook
    raise BackendUnavailable
BackendUnavailable

py27 installed: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.,apipkg==1.5,atomicwrites==1.2.1,attrs==18.2.0,configparser==3.7.1,contextlib2==0.5.5,coverage==4.5.2,enum34==1.1.6,execnet==1.5.0,flake8==3.6.0,funcsigs==1.0.2,futures==3.2.0,importlib-metadata==0.8,mccabe==0.6.1,mock==2.0.0,more-itertools==5.0.0,path.py==11.5.0,pathlib2==2.3.3,pbr==5.1.1,pluggy==0.8.1,py==1.7.0,pycodestyle==2.4.0,pyflakes==2.0.0,pytest==3.10.1,pytest-cov==2.6.1,pytest-fixture-config==1.4.0,pytest-flake8==1.0.3,pytest-shutil==1.4.0,pytest-virtualenv==1.4.0,scandir==1.9.0,six==1.12.0,termcolor==1.1.0,virtualenv==16.3.0,zipp==0.3.3
________________________________________________________________________________________ summary ________________________________________________________________________________________
ERROR:   py27: InvocationError for command /Users/jaraco/code/main/setuptools/.tox/py27/bin/pip install --exists-action w -e /Users/jaraco/code/main/setuptools (see /Users/jaraco/code/main/setuptools/.tox/py27/log/py27-2.log) (exited with code 2)

I suspect this is related to #1642 if not just another manifestation of the same issue.

@jaraco
Copy link
Member Author

jaraco commented Jan 25, 2019

In order to unblock this project, I was going to configure tox to downgrade pip after creating the environment, but tox only allows one install command, and this project has already used up that configuration option to work around another issue, and it's currently not possible to specify a pip version.

@jaraco
Copy link
Member Author

jaraco commented Jan 26, 2019

After getting a test failure from tox, I'm able to replicate the failure outside of tox by invoking the same command tox does (or similar):

setuptools master $ .tox/py27/bin/pip install -e .                                                                                                                                       
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Obtaining file:///Users/jaraco/code/main/setuptools
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
Exception:
Traceback (most recent call last):
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 315, in run
    resolver.resolve(requirement_set)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/resolve.py", line 131, in resolve
    self._resolve_one(requirement_set, req)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/resolve.py", line 226, in _get_abstract_dist_for
    req, self.require_hashes, self.use_user_site, self.finder,
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 382, in prepare_editable_requirement
    abstract_dist.prep_for_dist(finder, self.build_isolation)
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 149, in prep_for_dist
    reqs = self.req.pep517_backend.get_requires_for_build_wheel()
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_vendor/pep517/wrappers.py", line 71, in get_requires_for_build_wheel
    'config_settings': config_settings
  File "/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_vendor/pep517/wrappers.py", line 162, in _call_hook
    raise BackendUnavailable
BackendUnavailable

The problem, of course, is that the virtualenv doesn't have setuptools installed. Setuptools historically has relied on itself to build/install itself... but pep517 is breaking from the convention/expectation that '' is on sys.path.

@jaraco
Copy link
Member Author

jaraco commented Jan 26, 2019

the virtualenv doesn't have setuptools installed

I'm wrong about that. The virtualenv does have setuptools installed... it just doesn't show up with pip freeze (probably due to special handling).

@jaraco
Copy link
Member Author

jaraco commented Jan 26, 2019

If I update pep517._in_process to inject '' to sys.path in main(), the command no longer fails. That might also explain why python -m pep517.build . succeeds also - because in that mode of invocation, '' will be on sys.path.

@jaraco
Copy link
Member Author

jaraco commented Jan 26, 2019

This is particularly difficult to debug because of the subprocess invocation. If I put a breakpoint in the _in_process script to troubleshoot, it exits immediately because there's no pipes for stdin/stdout.

@jaraco
Copy link
Member Author

jaraco commented Jan 26, 2019

With lots of edits in the pip codebase, I managed to get a PDB prompt inside the _in_process, and here's what I see:

setuptools master $ .tox/py27/bin/pip install -e .                                                                                                                                       
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Obtaining file:///Users/jaraco/code/main/setuptools
  Installing build dependencies ... done
> /Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py(186)main()
-> if len(sys.argv) < 3:
(Pdb) import sys
(Pdb) sys.path
['/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/site-packages/pip/_vendor/pep517', '/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-KYbcw6/site', '/Users/jaraco/code/main/setuptools/.tox/py27/lib/python27.zip', '/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7', '/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/plat-darwin', '/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/plat-mac', '/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/plat-mac/lib-scriptpackages', '/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/lib-tk', '/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/lib-old', '/Users/jaraco/code/main/setuptools/.tox/py27/lib/python2.7/lib-dynload', '/usr/local/Cellar/python@2/2.7.15_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/usr/local/Cellar/python@2/2.7.15_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/usr/local/Cellar/python@2/2.7.15_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/usr/local/Cellar/python@2/2.7.15_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/usr/local/Cellar/python@2/2.7.15_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-KYbcw6/overlay/lib/python2.7/site-packages', '/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-KYbcw6/normal/lib/python2.7/site-packages']
(Pdb) import os
(Pdb) os.getcwd()
'/Users/jaraco/code/main/setuptools'
(Pdb) import setuptools
*** ImportError: No module named setuptools

Why is setuptools not available? I suspect it has something to do with the 'overlay' and the fact that the site-packages of the virtualenv isn't in sys.path.

@jaraco
Copy link
Member Author

jaraco commented Jan 26, 2019

~ $ ls /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-KYbcw6/normal/lib/python2.7/site-packages                                                                  
ls: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-KYbcw6/normal/lib/python2.7/site-packages: No such file or directory
~ $ ls /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-build-env-KYbcw6/overlay/lib/python2.7/site-packages                                                                 
wheel                  wheel-0.32.3.dist-info

So it seems something is creating a clean environment of dependencies in which the build can happen, containing only the dependencies declared in build-system.requires. That makes sense. As I was tracing the code, I didn't see where that happens.

@benoit-pierre
Copy link
Member

I've used pdb-clone for debugging such cases.

The build environment code is in pip._internal.build_env.

So setuptools is a particular case, since it provides its own backend, which mean that for supporting such a use-case, the package source directory must be added to sys path, and this must be done in pip.

@benoit-pierre
Copy link
Member

I can only reproduce the issue with Python 3 when tox-venv is not installed, which is also weird.

@jaraco
Copy link
Member Author

jaraco commented Jan 26, 2019

I can only reproduce the issue with Python 3 when tox-venv is not installed, which is also weird.

That's probably a pip bug in the build isolation code on a venv.

@benoit-pierre
Copy link
Member

No, it's because the versions of pip being used are different...

@jaraco
Copy link
Member Author

jaraco commented Jan 27, 2019

Following the acceptance of #1634, setuptools' own test suite now fails.

@jaraco jaraco changed the title Tests on Python 2.7 failing to run with BackendUnavailable since pip 19 Tests failing to run with BackendUnavailable since pip 19 Jan 27, 2019
@daa
Copy link
Contributor

daa commented Jan 28, 2019

As you have mentioned BackendUnavailable is raised because of ImportError: No module named setuptools and this happens because setuptools is not available in build environment. Speaking about details let's consider how pip builds wheel with pep517:

  1. Prepare build environment
  2. Install build requirements into that path
  3. From prepared build environment invoke build backend hook to get requirements to build wheel - get_requires_for_build_wheel
  4. Invoke build_wheel() hook from build environment

To find why setuptools is not available in the build environment inspite of it being in current working directory let's see how isolated build environment is prepared:

  1. Create temporary directory with structure resembling python directory layout
  2. Create sitecustomize.py (pip/_internal/build_env.py at line 79) which removes original paths from sys.path, adds system site directory and adds build environment site directory.

So step 2 is making setuptools from current directory unavailable for code run inside of build environment.

@pganssle
Copy link
Member

Presumably even though @daa's solution works, it means that we can't actually bootstrap our PEP 517, right? We will always be relying on the previously built versions of setuptools.

That will probably work for now, but this seems like it must be a bigger issue, and I think it's one we can't fix in this project, right? The issue is that we supply our own PEP 517 backend, and there's no way to declare that you depend on a PEP 517 backend that exists only in your local directory.

I see two ways forward with this:

  1. Fix this problem with pip, adjusting PEP 517 to allow this kind of situation if necessary.
  2. Create a new "PEP 517 bootstrapper" backend, that simply delegates all backend tasks to a backend that exists in your local directory, then start using that.

Of course with option 2 the PEP 517 bootstrapper still needs some way to build itself, and all PEP 517 build systems have this problem, so if I understand the problem correctly, the most robust solution is to pursue option 1.

Is there already a ticket tracking this in pip?

@ncoghlan
Copy link
Member

ncoghlan commented Feb 2, 2019

I think the discussionin Discourse is still going somewhere, but the direction it's now going is the direction you describe here: projects that need bootstrapping should add themselves to their own build-system.requires, and then the onus is on frontends to detect that and halt the infinite regress.

@pganssle
Copy link
Member

pganssle commented Feb 4, 2019

Thinking about this, even if pip --no-binary :all: switches to resolving self-referential source distributions with wheels, if they don't handle long-range cycles, we're still going to break --no-binary :all: if we depend on wheel as part of the PEP 517 build.

That basically means that we cannot have any dependencies in build-system.requires, and we'll need to vendor wheel, so I think we have three options:

  1. Specify 'setuptools' and 'wheel' in our build-system.requires and let pip fix --no-binary :all: or not, as desired.
  2. Opt out of PEP 517 forever, meaning that we will never be installable by a PEP 517-only frontend except as a wheel.
  3. Vendor wheel in our source tree and wait for PEP 517 to be extended to allow for self-bootstrapping builds.

I don't think number 3 is going to happen, so we're really choosing between options 1 and 2. I'm in favor of option 1, but we may want to give pip some fair warning that we'll be doing it, since it will likely cause at least some breakages.

@ncoghlan
Copy link
Member

ncoghlan commented Feb 4, 2019

Reading pypa/pip#6222 (and based on the other threads), I think the pip folks already agree that the current interaction between --no-binary :all: and PEP 517 needs work, so Option 1 seems like a reasonable choice at the setuptools level to me.

benoit-pierre added a commit to benoit-pierre/setuptools that referenced this issue Aug 12, 2019
Make it possible to use a more recent version of pip for tests.
benoit-pierre added a commit to benoit-pierre/setuptools that referenced this issue Aug 13, 2019
Make it possible to use a more recent version of pip for tests.
benoit-pierre added a commit to benoit-pierre/setuptools that referenced this issue Aug 22, 2019
Make it possible to use a more recent version of pip for tests.
benoit-pierre added a commit to benoit-pierre/setuptools that referenced this issue Oct 7, 2019
Make it possible to use a more recent version of pip for tests.
benoit-pierre added a commit that referenced this issue Nov 13, 2019
benoit-pierre added a commit to benoit-pierre/setuptools that referenced this issue Nov 15, 2019
Work around buggy pip detection code for "pip.exe install/update ...  pip ...".
commit 0d831c9
Author: Benoit Pierre <[email protected]>
Date:   Sat Aug 10 03:57:58 2019 +0200

    improve
benoit-pierre added a commit to benoit-pierre/setuptools that referenced this issue Nov 15, 2019
Work around buggy pip detection code for "pip.exe install/update pip ...".
mergify bot added a commit that referenced this issue Nov 15, 2019
@pganssle
Copy link
Member

So after the absurd amount of discussion on this, the final decision (around 9 months ago) was to modify PEP 517 to add backend-path for bootstrapping, but no front-ends are actually spec compliant as far as I can tell. pip doesn't seem to support it at all.

This makes it very difficult to install our own library from git. I suggest that we just go ahead with this configuration for now:

[build-system]
requires = ["wheel", "setuptools>=40.0"]
build-backend = "setuptools.build_meta"

We can continue to exclude pyproject.toml from the source distribution so that pip install setuptools --no-binary :all: will continue to work for now. Once pip implements this we can probably switch over to using backend-path.

@webknjaz
Copy link
Member

webknjaz commented Feb 4, 2021

So after the absurd amount of discussion on this, the final decision (around 9 months ago) was to modify PEP 517 to add backend-path for bootstrapping, but no front-ends are actually spec compliant as far as I can tell. pip doesn't seem to support it at all.

backend-path is now used on the main branch requiring pip >=20 or something like python-build.

@jaraco
Copy link
Member Author

jaraco commented Feb 26, 2021

I see two ways forward with this:

1. Fix this problem with `pip`, adjusting PEP 517 to allow this kind of situation if necessary.

2. Create a new "PEP 517 bootstrapper" backend, that simply delegates all backend tasks to a backend that exists in your local directory, then start using that.

After some consideration, I do think this project needs something like a bootstrapper backend, mainly to address the situation of a clean source checkout. In that situation, there is no metadata present. Setuptools currently works around that situation by requiring a separate copy of setuptools to build, which is okay, but also clumsy, because then when the sdist is built, it also declares a requirement on setuptools, even when it's not needed.

@jaraco
Copy link
Member Author

jaraco commented Feb 27, 2021

In the above referenced pull request, I figured out that Setuptools can supply its bootstrapping metadata in a separate metadata file. There's still no way to ever make Setuptools build properly on pip 19, so it will just have to break.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants