pip install of cartopy 0.18.0 fails when installing numpy at the same time

[mc-allen]

Description

I am provisioning a docker image using pip. In a single pip command, I installed a number of packages, including cartopy and numpy. This worked in versions prior to 0.18.0, and no longer works with 0.18.0.

Code to reproduce

In a docker image with vanilla python3 install and no pip packages installed, run

pip3 install --upgrade pip   && pip3 install --no-cache-dir cartopy==0.18.0 numpy

Traceback

    ERROR: Command errored out with exit status 1:
     command: /usr/local/pyenv/versions/3.7.6/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-cdx7ek5c/cartopy/setup.py'"'"'; __file__='"'"
'/tmp/pip-install-cdx7ek5c/cartopy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(cod
e, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-377te2k_
         cwd: /tmp/pip-install-cdx7ek5c/cartopy/
    Complete output (12 lines):
    Traceback (most recent call last):
      File "/tmp/pip-install-cdx7ek5c/cartopy/setup.py", line 43, in <module>
        import numpy as np
    ModuleNotFoundError: No module named 'numpy'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-cdx7ek5c/cartopy/setup.py", line 45, in <module>
        raise ImportError('NumPy 1.10+ is required to install cartopy.')
    ImportError: NumPy 1.10+ is required to install cartopy.
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Full environment definition

Operating system

Ubuntu 18.04
Python 3.7.6 installed via pyenv

Cartopy version

0.18.0

[QuLogic]

This is because pyproject.toml was removed; it, or some other form of that PEP, might come back in the future, but it depends on pip not having weird behaviour with it.

[akrherz]

Peanut gallery here. What's the issue with just having setup.cfg install/build require numpy? My present workaround to this is just to pip install numpy before pip install cartopy and it seems to work, but I have a blissful life on linux 64 bit :)

[dopplershift]

@akrherz That's a great idea, which is why numpy is already listed in requirements/defaults.txt, which is what's used by our setup.py. Unfortunately, and this is one of the reasons behind PEP-517/pyproject.toml, the way pip works is such that it only works for run requirements, not build requirements. It essentially doesn't register as available.

easy_install was a tool that worked around this in the days of yore, but it created more problems than it solved.

[nbren12]

I just ran into this problem too.

This is because pyproject.toml was removed; it, or some other form of that PEP, might come back in the future, but it depends on pip not having weird behaviour with it.

What was the motivation behind this change? It makes it quite challenging to use cartopy with projects that use poetry.

[dopplershift]

@nbren12 The commit log on 454a911 basically says it all--at the time at least (I've not tested since) pip, when using pyproject.toml, would helpfully ignore the numpy you already have installed and instead try to install the latest to use when building and installing your project.

[graingert]

@dopplershift if the version in install_requires matches the version in build-system requires then this won't be a problem. You'd want to pick the widest possible range of numpy versions with a stable ABI

Also a set of manylinux,macos and windows abi3 binary wheel would solve this for most users

[dopplershift]

I don't think it's so much an end-user issues as a developer one. If I install numpy 1.16 to test, I want cartopy to build against it. And if I've already installed numpy from conda, I sure as hell don't want it downloading something from pypi--especially an sdist. See #1264 (comment) and the surrounding discussion.

[graingert]

@dopplershift this will be fine if the build-system and install_requires numpy version ranges match, and span only a single numpy ABI. pip 20.4 will uninstall cartopy and install one with a compatible numpy version.

In fact you wouldn't even need a build time dep on numpy as you could include the .h files for the numpy ABI you supported in your sdist

[dopplershift]

Might be worth giving pyproject.toml another try then.

[nbren12]

Thanks for the explanation. It sounds like it is important to test against specific versions of numpy, and that you don't find pip
s interpretation of PEP-517 useful for testing purposes. Fair enough. However, instead of breaking pip installs for users, would it be possible to avoid using pip in the tests (e.g. modify them to use python setup.py install)?

This issues matters because "serverless" cloud python runtimes like AWS lambda and Google Cloud dataflow tend to support pip but not anaconda. Also, IMO anaconda has fallen a bit behind the curve in terms of dependency locking and other features that tools like poetry and pipenv provide.

[dopplershift]

IMO, because poetry/pipenv rely on Python's wheel standard, they're not as good for the scientific stack--they have no way to package non-Python dependencies. This causes problems where Shapely then builds and links to one copy of GEOS, and CartoPy links to another; if you try to then import those packages in the same script: BOOM. You don't have that problem with Anaconda because you just 'conda install geos` and those other packages depend on it.

At any rate, I'm not against trying to fix it. If someone contributes a pyproject.toml and it no longer creates problems, then I'm happy to merge.

[nbren12]

I totally agree about the difficulties with pip and non-python dependencies. Unfortunately, I don't think conda has kept up with some of the innovations of the broader community around lockfiles/deterministic builds and project management. I would rather use two package managers (apt-get for system libraries) and pip/poetry/pipenv for python dependencies than sacrifice determinism. conda install cartopy xarray fsspec scikit-learn zarr is great, but put that in a CI system and eventually some dependency update will break the build. In my experience with this stack, this happens about twice a month and takes 30+ minutes to debug each time. I would much rather spend 10 minutes up-front writing a dockerfile/shell script with poetry +apt-get if it is possible to achieve a deterministic build.

At any rate, I don't think it should be too hard to make cartopy PEP-517 compliant.

[graingert]

How often does numpy break ABI?

…

On Wed, 18 Nov 2020, 22:54 Noah D Brenowitz, ***@***.***> wrote: I totally agree about the difficulties with pip and non-python dependencies. Unfortunately, I don't think conda has kept up with some of the innovations of the broader community around lockfiles/deterministic builds and project management. I would rather use two package managers (apt-get for system libraries) and pip/poetry/pipenv for python dependencies than sacrifice determinism. conda install cartopy xarray fsspec scikit-learn zarr is great, but put that in a CI system and eventually some dependency update will break the build. In my experience with this stack, this happens about twice a month and takes 30+ minutes to debug each time. I would much rather spend 10 minutes up-front writing a dockerfile/shell script with poetry +apt-get if it is possible to achieve a deterministic build. At any rate, I don't think it should be too hard to make cartopy PEP-517 compliant. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1552 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADFATAE4CYNLEBAIJCFY5TSQRGDDANCNFSM4M2QEKLA> .

[nbren12]

Check out the draft PR I just opened (#1680). It shows that you can control the installed numpy by switching to python setup.py develop vs pip install ..

[dopplershift]

Thanks for sending in the PR--I'll check it out when I have a chance to sit down more fully on some cartopy work.

Regarding CI stuff: oh I'm well aware. I'm maintaining plenty of CI setups and the breakage caused in both PyPI and Conda stacks is exceedingly irritating, to the point where I recently rewrote one to make significant use of Dependabot. The difference for me is that I'm unwilling to deal with getting GDAL, GEOS, PROJ, netCDF, etc. installed and built consistently on Windows, so I rely on conda there--therefore my CI solution had to work with both (conda create -n myenv --file requirements.txt is your friend).