Install install_requires before running python setup.py bdist_wheel

[paugier]

Environment

• pip version: 19.0.1
• Python version: CPython 3.7

Description

pip executes python setup.py bdist_wheel before installing the packages listed in install_requires.

For a simple package depending on numpy, using numpy in setup.py but such that python setup.py egg_info works fine without numpy, it shouldn't be necessary to use setup_requires (and to have a setup.py compatible with setup_requires, which is not easy).

From https://pip.pypa.io/en/latest/reference/pip_install/#installation-order

Although the new install order is not intended to replace (and does not replace) the use of setup_requires to declare build dependencies, it may help certain projects install from sdist (that might previously fail) that fit the following profile:

• They have build dependencies that are also declared as install dependencies using install_requires.
• python setup.py egg_info works without their build dependencies being installed.
• For whatever reason, they don’t or won’t declare their build dependencies using setup_requires.

For example, for this setup.py

import sys
from setuptools import setup

if "egg_info" in sys.argv:
    setup()
    sys.exit()

import numpy as np

setup()

and this setup.cfg:

[metadata]
name = irequires

[options]
install_requires =
    numpy

the command pip install . fails with

Processing /home/users/augier3pi/Dev/try_py_dependencies/irequires
Collecting numpy (from irequires==0.0.0)
  Using cached https://files.pythonhosted.org/packages/3d/10/62224c551acfd3a3583ad16d1e0f1c9e9c333e74479dc51977c31836119c/numpy-1.16.0-cp37-cp37m-manylinux1_x86_64.whl
Building wheels for collected packages: irequires
  Building wheel for irequires (setup.py) ... error
  Complete output from command /home/users/augier3pi/tmp/tmp_venv/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/tmp/pip-req-build-_0f8h24y/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-6n4locuo --python-tag cp37:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-req-build-_0f8h24y/setup.py", line 10, in <module>
      import numpy as np
  ModuleNotFoundError: No module named 'numpy'
  
  ----------------------------------------
  Failed building wheel for irequires
  Running setup.py clean for irequires
  Complete output from command /home/users/augier3pi/tmp/tmp_venv/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/tmp/pip-req-build-_0f8h24y/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" clean --all:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-req-build-_0f8h24y/setup.py", line 10, in <module>
      import numpy as np
  ModuleNotFoundError: No module named 'numpy'
  
  ----------------------------------------
  Failed cleaning build dir for irequires
Failed to build irequires
Installing collected packages: numpy, irequires
  Running setup.py install for irequires ... done
Successfully installed irequires-0.0.0 numpy-1.16.0

Expected behavior

pip should install numpy before executing python setup.py bdist_wheel so that this simple setup works fine as explained in https://pip.pypa.io/en/latest/reference/pip_install/#installation-order.

Note

Listing packages both in install_requires and setup_requires don't work well. See pypa/setuptools#209 and pypa/setuptools#391

[paugier]

Note that we still get exactly the same error even by adding setup_requires in setup.cfg,

[metadata]
name = irequires

[options]

install_requires =
    numpy

setup_requires =
    numpy

[RonnyPfannschmidt]

setup_requires does not match with PEP-517, so thats not supportable for pip
install_requires lists the things a package needs when it was installed, so thats explicitly distinct from what it needs for building

if you need numpy for the build of a package in isolation, use the PEP-517/518 metadata or set up a custom build without isolation

[paugier]

Few days ago, I opened issue #6144 on PEP-517. I think this PEP (isolated build) is a problem for some packages (and also for compatibility with conda) so many packages won't be able to use it.

It's really a problem that the very simple package described in the issue cannot be installed with just one simple pip command.

It would be so simple for pip! The numpy wheel is ready.

I really don't understand what would be the problem with installing the wheels as soon as they are ready and not to build all wheels and then installing them.

Otherwise, why is there the sentence I quoted in https://pip.pypa.io/en/latest/reference/pip_install/#installation-order ?

[RonnyPfannschmidt]

build isolation builds things in a completely pristine environment, - its a positive key property for repeatability and comprehension

a key need is not having random things affect the state, which seems to be your desire to get as a feature - my personal opinion on that is -- sorry no go, please try again with something reliable

[paugier]

I really understand that isolation build is very good and useful for most packages.

I also think that many good things in Python have been done before without isolation build and that it's not a good idea to force people to use isolation build if you want to impose a new standard (PEP 518, pyproject.toml), but anyway.

You also have to understand that there are reasonable reasons to avoid isolation builds (for example compatibility with conda [which is not a small subject in numerical / scientific / data Python], or using packages for which there are no wheels on PyPI, as mpi4py).

So simple things have to work without build isolation and pyproject.toml. In particular, things like what I describe in this issue should work. At least, there should have a pip option to get this nice behavior (which is not crazy).

It really seems that it is not even technically difficult from the pip point of view.

What is the drawback of installing wheels as soon as they are ready? It seems to me that pipenv does it (at least tox-pipenv). Anyway, isolation build won't become the default for package without pyproject.toml.

What is the advantage of building the wheels from all collected sdist with the old environment?

Also, it would be nice to answer about the sentence at the end of the paragraph https://pip.pypa.io/en/latest/reference/pip_install/#installation-order

Is it just wrong? If it is wrong, should it be removed from the documentation?

Moreover, why was it included in the official pip documentation if it is a such a terrible practice to rely on non-isolated builds and on the evolution of the environment during the pip command execution?

[pradyunsg]

The guarantee is about installation order; not build order. It does not suggest that it would install a install_requires entry before a setup.py is executed.

I suggest you use PEP 518, or disable isolation with --no-build-isolation, if you have a build-time dependency.

[paugier]

It does not suggest that it would install a install_requires entry before a setup.py is executed.

Yes it does! 🙂

I'm sure that the last paragraph of https://pip.pypa.io/en/latest/reference/pip_install/#installation-order is really about this issue. From what is written (see my quote in the first message of the issue), the example in this issue should work. It's clear.

I have to repeat that PEP 518 (pyproject.toml) implies PEP 517 (build isolation) and that build isolation is not compatible with conda environment (so pyproject.toml won't be considered in conda env, so the problem won't be solved in conda env) and cannot be used for many projects (for example all projects using mpi4py in their setup.py).

Using --no-build-isolation leads to the same result as without pyproject.toml, i.e. pip is not able to install the example package in a fresh environment.

So unfortunately, pyproject.toml is not a nice universal solution.

A simple solution would be that pip installs each wheel that has to be installed just when it is ready (before the next build) and not at the end, when all wheels have been built. What is the problem of doing that?

[dstufft]

install_requires are runtime dependencies, not build time, and pip doesn't install them prior to trying to build the package in question.

[paugier]

@dstufft

I know. I try to tell you that there is a problem with the new standard (pyproject.toml / build isolation) of how to declare build dependencies and that it would be good if what is written in the official documentation works.

From https://pip.pypa.io/en/latest/reference/pip_install/#installation-order

Although the new install order is not intended to replace (and does not replace) the use of setup_requires to declare build dependencies, it may help certain projects install from sdist (that might previously fail) that fit the following profile:

• They have build dependencies that are also declared as install dependencies using install_requires.
• python setup.py egg_info works without their build dependencies being installed.
• For whatever reason, they don’t or won’t declare their build dependencies using setup_requires.

If there a problem with my English? Why don't you understand me? Isn't it clear that if we take seriously what is written in this paragraph, the example in this issue should work.

• the build dependency (here, numpy) is declared in install_requires.
• python setup.py egg_info works without numpy
• For whatever reason (in particular because it leads to problems to declare the same package in install_requires and in setup_requires, as already explained above), we don't use setup_requires

I have to admit that I am sad about how my remarks (this issue and issue #6144) are treated. In scientific and data applications, conda is used a lot. My reasoning is that it is not a great thing that the Python community is split in terms of installation tools, so I tried a lot to be also able to work without conda. I thought that pip could also be adapted for our applications. But now, I start to see how concerns from my community are treated by pip core developers.

[pfmoore]

Maybe the wording of that document is a little unclear. But the principle is simple:

• Build dependencies are declared via setup_requires (or since PEP 518, via requires in pyproject.toml).
• Runtime requirements (or "install dependencies") are declared via install_requires.

Those terms are possibly a little confusing (they confuse me!) but they have been standard for many, many years.

"They have build dependencies that are also declared as install dependencies using install_requires" means that the project declares that foo is both a build dependency and a runtime dependency (by including it in both pyproject.toml and install_requires). There's no implication here (or anywhere else) that a runtime dependency will automatically be available as a build dependency. Since PEP 518 support in pip introduced build isolation, clearly understanding the distinction is crucial if you expect your build step to work properly.

You say that "For whatever reason (in particular because it leads to problems to declare the same package in install_requires and in setup_requires, as already explained above), we don't use setup_requires". That's fine - setup_requires is now obsolete, precisely because it had issues (the build requirements were downloaded by setuptools and not by pip, for example). But that means you have to use pyproject.toml and the requires key, not that you can expect pip (or any other tool) to automatically infer your build dependencies¹. But before pyproject.toml, you'd have either had to use setup_requires, or rely on some sort of undocumented and accidental behaviour.

I have to admit that I am sad about how my remarks are treated

All I can say is that we're trying. Personally, I'm struggling to understand what you want and why the approaches that have been suggested to you are so unacceptable. I will say that "because conda needs pip to work like this" is not a compelling argument, at least to me. While I'm 100% in favour of better interoperability between conda and pip, I don't think that will be achieved by simply making pip change "to work with conda". What's needed is a better understanding of the shared requirements of the two ecosystems and how both tools can compromise to make things work better - while still remaining true to their core goals and user communities.

¹ It's possible that the setuptools backend is introspecting your setup.py and converting setup_requires requirements into build requirements as part of the get_requires_for_build_wheel hook. But I don't understand that code so I might be wrong, and even if I'm not that's something specific to setuptools, not anything that pip is involved with.

[pradyunsg]

Let's improve the wording in the documentation. I think that's the reason for the confusion in this case.

[jdemeyer]

I think that the documentation at https://pip.pypa.io/en/stable/reference/pip_install/#installation-order is pretty clear: it describes a use case that used to work completely fine with the "classical" (not based on wheels) installation. But the same use case no longer works with the "new" (first build wheel, then install from wheel) installation. Unfortunately, pyproject.toml forces the "new" installation, breaking such packages.

See also the discussion at https://discuss.python.org/t/support-for-build-and-run-time-dependencies/1513

[jdemeyer]

FYI: the feature of installing install_requires of before building the package was introduced in #2616, issue #2478. It's clear to me that this was meant to work the way that we're expecting it to work.

Now, I totally understand that it's difficult to support this feature with pyproject.toml. But then there needs to be a replacement for that feature and for the moment there isn't.

[cjerdonek]

Just to clarify, does this also not work with --no-use-pep517? I thought --no-use-pep517 was the escape hatch to restore the old behavior, so is the bug that this isn't working? Also, is 19.0.1 the first version where this stopped working as the reporter wanted?

[jdemeyer]

@cjerdonek: The problem is not so much PEP 517, the problem is really bdist_wheel. With PEP 517, bdist_wheel is mandatory and the build fails if the build-and-run-time dependencies are not specified in either pyproject.toml or setup_requires. Without PEP 517, the installation is first done with bdist_wheel, which fails (just like with PEP 517). But in this case, pip falls back to the legacy install mode which does succeed.

So nothing got broken for now (in the sense that packages that used to install still install). But that's only because pip has a fallback to the old way of installing packages. If that fallback ever gets removed or pyproject.toml becomes mandatory, then this will actually break.

[cjerdonek]

Yes, that was the reason for the fallback: to make sure things could continue working and, I assume, to get a better understanding of what remaining cases may need further thought or discussion (of which this issue is an example). I'm certain there will be ample discussion before the fallback is removed (see #6334 for the anchor issue on this).

In asking my question, I just wanted to confirm 19.0.x did have a work-around that was working and wasn't breaking for you entirely.

[ax3l]

I have exactly the same problem as the OP for projects that depend on PyPI's "cmake" package (in install_requires) to be build.

Since the install of the "cmake" package finishes "too late", a user will not build a wheel (unless executing "pip install ." twice) for my source packages, although it could if cmake were properly installed before starting to build. Here is the cumbersome logic that is executed:

$ pip install .
  Created temporary directory: /tmp/pip-wheel-tjx61hzn
  Running setup.py bdist_wheel for openPMD-api ...   Destination directory: /tmp/pip-wheel-tjx61hzn
  Running command python -u -c "..." bdist_wheel -d /tmp/pip-wheel-tjx61hzn --python-tag cp36
  running bdist_wheel
  running build
  running build_ext
  Traceback (most recent call last):
   # ...
    File "/tmp/pip-req-build-sa81uv0f/setup.py", line 53, in run
      self.build_extension(ext)
    File "/tmp/pip-req-build-sa81uv0f/setup.py", line 61, in build_extension
      ", ".join(e.name for e in self.extensions))
  RuntimeError: CMake 3.11.0+ must be installed to build the following extensions: openpmd_api
error
  Failed building wheel for openPMD-api
  Running setup.py clean for openPMD-api
  Running command python -u -c "..." clean --all
  running clean
Failed to build openPMD-api

Installing collected packages: cmake, openPMD-api
# CMake: Why so late?

  changing mode of /home/axel/miniconda3/bin/cmake to 755
  changing mode of /home/axel/miniconda3/bin/cpack to 755
  changing mode of /home/axel/miniconda3/bin/ctest to 755
  Created temporary directory: /tmp/pip-record-uyk6zbxg

  Running setup.py install for openPMD-api ...     Running command python -u -c "..." install --record /tmp/pip-record-uyk6zbxg/install-record.txt --single-version-externally-managed --compile
    running install
    running build
    running build_ext

  # now it finds the now usable cmake binaries ...

[ax3l]

Ok, I should learn to read all comments.

Indeed, following PEP-518 and just adding a pyproject.toml file with

[build-system]
requires = ["setuptools>38.6", "wheel", "cmake>=3.11.0,<4.0.0"]

context will be picked up by pip and solves the issue.

[pradyunsg]

A clarification has been added to the relevant section (see #6604).

Closing this since there isn't much else actionable here since we mostly just needed a clarification here.