RFC: Python 2 support

[effigies]

As I'm sure we're all aware, CPython 2 support ends on January 1, 2020, and a number of projects have committed to cutting off Python 2 support in the coming year, some already having effected the change in master. In particular, numpy, scipy, scikit-learn and scikit-image, which nibabel either depends on or is often used in conjunction with, have all committed to ending their Python 2 support.

We cannot realistically extend our support of Python 2 much farther, even if we want to. I would like to propose the upcoming 2.4 series as the last series with any Python 2 support at all, and commit to bug-fixes in 2.4.x until January 2020. This is a short timeline, but hopefully piggy-backing on the zeitgeist will help keep it from being too jarring.

I am open to someone else maintaining the 2.4.x branch for a short while after January 2020, but I think it should be short.

Assuming we settle on this scheme, nibabel 3.0 will remove all Python 2 support, and I'll do my best to ensure that the PyPI metadata will require Python 3+, so that Python 2's pip will not attempt to install nibabel>=3.

@nipy/core @nipy/team-nibabel @GaelVaroquaux @pauldmccarthy @afni-rickr

[yarikoptic]

What will be the minimal python3 version you are aiming for, or in other words what feature(s) of python3 line you would like to start using. here we collected some notable new features with python3 versions where they were introduced. May be there is some nice concise overview already of a similar kind?

[effigies]

I think my overall feeling about nibabel is that we should be pretty conservative and support as far back as we can, but outlasting CPython itself is not a good use of effort. Python 3.4 is supposed to have a final release this month, which means it will predecease 2.7 by a little bit.

So my inclination would be to support 2.7 and 3.4 through nibabel 2.4.x, and jump to 3.5+ for nibabel 3.0. We won't get f-strings or data classes, but I don't see those as particularly critical to nibabel's development. The main thing I know of that 3.6+ would buy us that makes me look forward to dropping 3.5 is insertion-ordered dictionaries obviating the need for OrderedDict, but I think maintaining compatibility outweighs that desire.

[matthew-brett]

Sounds reasonable to me. I wonder whether we'll need Python 2 support (via 2.4) for a year or so after Python 2 end-of-life though. I doubt that would be a lot of extra work.

[effigies]

That's probably true. Mostly will mean needing to take the maint/* branches seriously.

[yarikoptic]

Do we need maint/ prefix? I would just have release branches like 2.3.x , 2.4.x

[effigies]

It's what we've been doing. At least for 2.4.x, let's continue with this trend, following the principle of least surprise. I don't have a strong commitment to the form, though, so if the population vector points in that direction, I'm okay with dropping the maint/ prefix at 3.0.x.

[yarikoptic]

Ah right, missed that it already exists - all good then, not worth disturbing the force

[effigies]

Oh, minor amendment: Forgot that by the schedule put forward in #734, 2.5 will be due out in June (give-or-take). So that would be the final Python 2-supporting series.

[raamana]

Do you guys think knowing the usage extent of nibabel with python < 3.0 would help? If knowing that (and why) would help this decision making process, I think pypinfo can help us here: https://github.com/ofek/pypinfo

Chris, regarding dict in python 3.5, please note:

"The order-preserving aspect of this new implementation is considered an implementation detail and should not be relied upon (this may change in the future, but it is desired to have this new dict implementation in the language for a few releases before changing the language spec to mandate order-preserving semantics for all current and future Python implementations; this also helps preserve backwards-compatibility with older versions of the language where random iteration order is still in effect, e.g. Python 3.5)."

https://docs.python.org/3.6/whatsnew/3.6.html#new-dict-implementation

[effigies]

I don't know the usage extent, but I believe that PyMVPA depends on nibabel and is still Python 2-only.

[raamana]

I just quickly ran pypinfo, and here are the download stats for nibabel for the last 800 days (~2 years). I think there is some noise in these numbers (and some repeat downloads I believe), but the their fractions should help us give a rough idea of usage. So python 2.7 and python 2.6 account for about 34% usage.

$ 14:08:47 miner ~ >>  pypinfo -sd -800 nibabel pyversion
Served from cache: False
Data processed: 589.72 GiB
Data billed: 589.72 GiB
Estimated cost: $2.88

| python_version | download_count |
| -------------- | -------------- |
| 3.6            |        362,950 |
| 2.7            |        243,117 |
| 3.5            |         70,889 |
| 3.4            |         21,275 |
| 3.7            |         14,280 |
| 2.6            |            360 |
| 3.3            |             96 |
| 3.2            |             21 |
| 3.8            |              6 |
| None           |              3 |
| Total          |        712,997 |

$ 14:10:26 miner ~ >>  

[yarikoptic]

It might be just that they are just still tested against and instated using pip. And there would be a sudden drop as those projects move on ;-) I really doubt that any recent 2.6 download had a human behind it. 2.7 though IMHO will still see use for a year or two on systems where 2.7 is the default one (majority of released Linux distributions ATM? ;-))

[raamana]

True - it is possible a non-trivial portion of the downloads for python 2.7 are from CI testing matrices. That doesn't necessarily imply they are also tested for and support python 3+.

It would have been awesome if we had a way to mine the pypi index to find out all the packages that declared nibabel as a dependency, and see how many we would break by orphaning python 2.7. Usually this link shows it, but perhaps only admins can see it.

[effigies]

Usually this link shows it, but perhaps only admins can see it.

No, GitHub's not very good about following Python dependency graphs. Many actual dependencies will fail to be detected.

[yarikoptic]

I don't know the usage extent, but I believe that PyMVPA depends on nibabel and is still Python 2-only.

Thanks for keeping it in mind! but don't ;) There is WiP: PyMVPA/PyMVPA#525 and if we don't get it done by the time nibabel-py3-only comes about, it is Ok ;) Bringing the codebase, pieces of which were still coded for then modern python 2.4, is so much "fun" ;)

[raamana]

Here is a counter suggestion: instead of supporting older versions of nibabel, we could actually help upgrade the dependent packages (with exceptions) to Python 3! that way our collective energy is better spent moving everybody to the latest than just stuck with “fun” task of maintaining older versions..

[raamana]

Some packages will have structural limitations who won’t be able to upgrade, who would have to stick with older stable releases of nibabel (although this fraction may be negligible)

[effigies]

@raamana Helping projects upgrade to Python 3 seems like a good and noble thing, and if we had a hackathon (i.e., limited timeline) environment, that would be a pretty cool contribution. I'm worried about committing to too many things, though I'm happy to provide incidental help as people work toward it. I'm not sure how to advertise that effectively, though. Possibly this year's OHBM hackathon?

I think our biggest legacy users are likely to be people in scientific computing environments that use old CentOS or similar. That was the cohort that submitted 2.6 compatibility patches after we officially dropped support. So probably the best thing we can do for people in that situation is to make sure that Python 2 users can't upgrade to nibabel 3.0 through normal installation procedures (pip, conda, apt-get, yum).

@ignatenkobrain Are you still the yum packager? And @TheChymera were you packaging for Gentoo?

[TheChymera]

Yes, I am packaging stuff for Gentoo. The Gentoo philosophy is to not break things unless they're broken, so python2* support will remain available for as long as any significant amount of ppl will be using it. Currently ~1500 of the Python packages we provide lack 3.* support in upstream, so the plug will not be pulled anytime soon.

Regarding nipype in particular: each time I version bump it, the test suite is run for all selected python implementations, so as long as we mark it as supporting py2* I will be able to spot py2* bugs and report them, and for the reasons mentioned above, py2* will likely remain selectable for a very long time.

Personally, I have moved to py3* on account of blender not supporting py2*, the only reason I continue to test things for py2* is because Gentoo makes it so easy.

[effigies]

So Debian resolves this by having python-* python3-* packages, so the easy thing there is to not try to build packages for python-nibabel for 3.0+, while python3-nibabel can just keep going. Is that similar over in Gentoo?

[TheChymera]

Dependency management on Gentoo can be as fine-grained as it needs to be, depending on whether I do something or nothing one of the following scenarios will happen (but I'm ok with whatever you do):

1. >=nipype-2 supports <nibabel-3 and I do not bother to do slotting:
   in this case people who select both python2.7 and python3.* will be able to use the latest nipype with the latest nibabel which supports both python versions.

2. >=nipype-2 supports <nibabel-3 and I bother to do slotting:
   in this case people who select both python2.7 and python3.* will use the latest nipype with the latest nibabel when they invoke the python3.* interpreter and the latest nipype with the latest nibabel which supports python 2 when they invoke the pthon2.* interpreter.

3. >=nipype-2 does not support <nibabel-3 and I have to do slotting:
   in this case people who select both python2.7 and python3.* will use the latest nipype with the latest nibabel when they invoke the python3.* interpreter and the latest nipype which supports nibabel which supports python 2 with the latest nibabel which supports python 2 when they invoke the pthon2.* interpreter.

4. >=nipype-2 does not support <nibabel-3 and I have to but don't do slotting:
   in this case people select python3.* only, or get stuck with an old nipype and nibabel, or implement slotting themselves :)

Note that none of these limitations affect people who only select a subset of the python3.* options.

[TheChymera]

but since nibabel won't continue to develop their py2 compatible branch, I don't think slotting is a sustainable feature.

[effigies]

Not sure if it affects your thinking here, but nipype 2.0 will be Python 3.6+, but the 1.x series is likely to continue for a while. I don't know if there will be a point at which 1.x also drops 2.7 support, or if that will simply be when 1.x development is finally abandoned.

[TheChymera]

That simply means that the problem disappears for >=nipype-2 and the aforementioned cases apply to <nipype-2 only, which means that slotting can be introduced here as well, if I want to... Will <nipype-2 just be bugfixes or include backports?

[raamana]

Chris @effigies - I don't mean to suggest we take any responsibility/commitment. I just mean to suggest we spend our time providing guidance: link to existing 2to3 How-Tos and tutorials, be a source of nipy-related issues/solutions, and suggest how to approach a big transition/port (if any). And in some truly subtle/complicated cases (likely rare), we can certainly fork/debug/port it.

Hackathons would be a perfect opportunity to spread this activity - in fact, I might suggest this as part of TrainTrack (adds to guidance as above).

[raamana]

But I also see your point on the trade-off: depending on the number of requests for help, and how small the change is (inside nibabel) to support python 2, it might be more efficient to fix it in nibabel, and encourage them migrate them on their own. Perhaps other core and long-term contributors of nipy ecosystem can weigh in here.

[effigies]

@TheChymera Nipype 2 is very nearly a complete rewrite with a new architecture, so backports aren't likely to apply. Once it's released, 1.x will probably drop to mostly bug fixes.

[TheChymera]

Hm, ok, so I'm going to continue testing for all Python versions currently marked as compatible, until you drop one and it fails and I remember you had told me here that you would drop it. Starting then I will drop the support, which will limit Gentoo users who concomitantly use python 2.7 and a more recent version to <nibabel-3 and <nipype-2 (latest common denominator). I can additionally make it so that even when multiple interpreters are present, each python interpreter gets to use the latest versions of the libraries compatible with it --- if enough people turn out to need this and open an issue/bug.

[effigies]

Sounds completely reasonable to me. Thanks!

[effigies]

Just a quick note that numpy support for 3.5 began in 1.10, so that will be the new minimum numpy when we drop 2.7 and 3.4. Anything earlier and we'd need to build wheels for CI tests, which doesn't seem worth it.

[effigies]

In case anybody on this thread isn't subscribed to nibabel in general, #787 is the PR for removing support for Python < 3.5.1. It also bumps the minimum numpy version to 1.12 and drops nibabel.py3k in favor of numpy.compat.py3k.

Any concerns? I assume Debian, Gentoo and Fedora (cc @sanjayankur31) are all packaging much newer numpys anyway.

[effigies]

Python 2 support has officially elapsed. If someone is interested in continuing to maintain the 2.5.x series, please get in touch.