Always use cythonized files in sdist.

[pelson]

Rationale

In #1258 @scoder suggested that we should consider making recompilation of pyx files optional. I can see some pros/cons to this approach, but it does mean that Cython is removed as an installation dependency, and would close #1258. The approach I've taken mimics numpy's - it only recompiles the Cythonized c++/c if we are not in a sdist (i.e. there is no PKG-INFO).

Implications

• Changes to the pyx files will be ignored when running setup.py IFF there is a PKG-INFO file at the same level as the setup.py (no for github source, yes for sdist).
• sdist now need to ship the Cythonized files as well as the pyx.

[QuLogic]

How do we ensure that the Cythonized files are included in the sdist?

[QuLogic]

License header failures though.

[pelson]

How do we ensure that the Cythonized files are included in the sdist?

python setup.py sdist should build and include them, no?

If it didn't, the sdist would not be installable (as it would automatically replace source.pyx with source.cpp, but the file wouldn't exist).

Unfortunately this is having issues with py27 and the compilation step.

[QuLogic]

Ah yes, you're right about sdist.

It's not Python 2; it's Python 2 with old Cython and Python 3 with old-ish Cython (which seems to be triggering the error in #1253.)

[pelson]

It's not Python 2; it's Python 2 with old Cython and Python 3 with old-ish Cython (which seems to be triggering the error in #1253.)

But why with this change, and not before... must be something odd with the way we setup the environment in travis.

[pelson]

Interesting, a successful log has the following:

$ pip install --no-deps .
Processing /home/travis/build/SciTools/cartopy
  Installing build dependencies ... done

No further useful diagnostics, but I wonder if a newer version of cython is being installed even though Cython is already installed.

[QuLogic]

That's also there in the failing build, though. A diff on the conda list output doesn't seem too illuminating.

[pelson]

That's also there in the failing build, though.

But I removed Cython from pyproject.toml's build_system item.

[QuLogic]

Ah right.

The CRON job also started failing yesterday and I thought it was the same error, but it was actually the NumPy ufunc thing.

[pelson]

Future-proofing for Cython 3.

[QuLogic]

I think we need to pass --no-build-isolation so it doesn't try to install new build deps. This might also fix the ufunc thing if we were also building against a newer numpy than is actually installed?

[pelson]

I think we need to pass --no-build-isolation so it doesn't try to install new build deps. This might also fix the ufunc thing if we were also building against a newer numpy than is actually installed?

Confirmed. An example log of pip installing build dependencies, even if they are installed (with decent stdout to see what is going on): https://circleci.com/gh/SciTools/cartopy/1277?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

Is there an issue somewhere in pip? scipy/scipy#8734 looks somewhat related. IMO It is quite unexpected that pip should re-install the build dependencies, even if they are already installed.

Needs updating

[pelson]

I just added a commit with the following description:

pyproject.toml was removed. It can be returned once pip no longer re-installs packages that are already installed. For example, if numpy is in [build-system]->requires, and numpy is already installed in the environment that is running pip, the existing version of numpy should be used to build cartopy, not the latest version found on pypi.

This should serve as a reference should we get future PRs adding a pyproject.toml - the situation must change with pip before cartopy will accept a pyproject.toml again. I fully recognise that this means that users who pip install cartopy will error out unless they pip install numpy before-hand. Unfortunately this is the state of the python packaging tooling, and is not something we can work around.

[QuLogic]

This should work to support 0.28 (as it seems there's no 0.29 for Python 3.5):

diff --git a/lib/cartopy/trace.pyx b/lib/cartopy/trace.pyx
index c1d8d5a..065946a 100644
--- a/lib/cartopy/trace.pyx
+++ b/lib/cartopy/trace.pyx
@@ -170,7 +170,7 @@ cdef class LineAccumulator:
                                                &geoms[0], geoms.size())
         return geom
 
-    cdef list[Line].size_type size(self):
+    cdef size_t size(self):
         return self.lines.size()

[pelson]

AppVeyor is queued, yet passed on the new SciTools one at https://ci.appveyor.com/project/SciTools/cartopy/builds/21631645/job/w1m0vd7bd74eo1ry.

[pelson]

@QuLogic - would you mind merging when happy with the changes? I don't think the AppVeyor results are going to come back (I've started fixing #1221 and am afraid I've caused this to happen accidentally).

With this merged we unblock the CI, have Cython 0.28 compatibility, and avoid needing Cython for users to install the sdist from pypi. 👍

[QuLogic]

I would keep the pins on Travis and AppVeyor, as I specifically put them in there to ensure it continued to work (even if that ended up being broken...)