Tracking issue for manylinux2014 rollout

[brainwane]

Now that PEP 599 has been accepted to adopt the manylinux2014 standard, we need a tracking issue for implementation (I'm reusing structure and prose from #179).

The dates are guesses but would (as I figure) help us get all the infrastructure in place well before Nov 2020 when CentOS 6 (supported in manylinux2010) will go EOL, and in time for the August 2020 date that @gunan asked for; please argue with me. ;-)

There are a number of further steps needed before folks can actually make use of that new baseline, and the order of operations matters (since building manylinux2014 wheels isn't very useful if installers won't install them):

Prep (ideally done by 2 September 2019):

• Gather personnel: tentatively strategize re: who's available to review & merge code, especially for repositories where maintainer time is scarce, re: who will make a build environment, and re: who will do testing or build testing infrastructure
• Curate initial list of test cases regarding build environment, target Linux distributions, specific Python packages to be built, presence/absence of specific libraries

Essential client support (ideally done by 15 September 2019):

• Update pip 19.x to install manylinux2014 wheels
  • Update packaging.tags to detect manylinux2014 compatibility (Add support for manylinux2014 packaging#186)
  • New release of packaging (19.2)
  • Add manylinux2014 support to pip._internal.pep425tags (Add manylinux2014 support pip#7102)
  • (Optional) Replace pip._internal.pep425tags w/ packaging.tags (Replace pep425tags w/ packaging.tags pip#6908)
  • New release of pip (19.3 due out in week 41, so, by 19 October)

Enable publication of manylinux2014 wheel archives (ideally done by 30 October 2019):

• Update Warehouse (PyPI) to accept manylinux2014 wheel uploads (Allow manylinux2014 wheel upload pypi/warehouse#6684)
• Update auditwheel to emit manylinux2014 wheel archives (manylinux2014 policy auditwheel#192)
  • Update auditwheel to allow new architectures introduced with manylinux2014 (Add support for new architectures in manylinux2014 auditwheel#202)
  • Pre-release of auditwheel (3.0.0.0rc1 Release 3.0.0 auditwheel#205)
  • New release of auditwheel (3.0.0 Release 3.0.0 auditwheel#205)
• Provide a manylinux2014 build environment, probably a Docker image (manylinux2014 images #348)
  • create quay.io repositories
• Confirm that twine will upload manylinux2014 wheel archives

Management of transition from manylinux1 and manylinux2010 (ideally done by 15 December 2019):

• Update https://packaging.python.org/specifications/platform-compatibility-tags/ to cover manylinux2014, with the caveat that it doesn't actually have broad tooling support yet, and to explain (per @zackw's note) pros & cons of switching to manylinux2014, e.g., future security patches, old Linux distributions new wheels won't work with, etc.
• Reword the "not yet widely supported" caveat from the platform compatibility tagging page on PyPUG

Additional projects to consider once core capability support has rolled out (ideally done by 1 March 2020):

• Resolve Why does warehouse allow linux_armv6l and linux_armv7l wheels? pypi/warehouse#3668
• Update pipenv to install manylinux2014 wheels (needed due to pipenv vendoring pip, so the support depends on the version of pipenv being used, rather than the version of pip in the host or target environments)
• Option to target manylinux2014 instead of manylinux2010 in cibuildwheel (Add support for manylinux2014 cibuildwheel#216, Add manylinux2014 x86_64/i686 support cibuildwheel#215)
• Option to target manylinux2014 instead of manylinux1 or manylinux2010 in multibuild
• Distinguish manylinux2014, manylinux2010 and manylinux1 images in dockcross
• Develop list of further relevant helper utilities and update them
• ... any other suggestions?

Tagging @di as the PEP 599 author, @pfmoore as the BDFL-Delegate who requested further planning on rollout, and @ewdurbin from the PyPI/PSF side of things.

[trishankatdatadog]

Minor comment, and probably too late now that the PEP has been accepted, but I feel it's relevant to avoid confusion: could we call it manylinux2024 (when Centos 7 actually expires) instead of manylinux2014 (sounds old and deprecated)?

[cjerdonek]

Regarding this task:

Replace pip._internal.pep425tags w/ packaging.tags

Is switching pip to use packaging.tags really a blocker formanylinux2014? @di's manylinux2014 patch was super tiny, so it seems like something similar could be done to pip's pep425tags for the short-term. I'm just concerned because it seems like switching pip to use packaging.tags is relatively a much bigger task, with things to do that aren't related to the manylinux2014 changes. (These additional tasks aren't currently reflected in the checklist above.) So I'd rather not rush the switch if there isn't really a need to. With this approach, switching to packaging.tags could be done for the pip release after the next.

[di]

Is switching pip to use packaging.tags really a blocker for manylinux2014?

No, definitely not, just figured it'd be nice to roll this in together since it seems like both are being worked on at the same time.

[di]

I just merged pypi/warehouse#6684, manylinux2014 wheels can now be uploaded to PyPI.

[xavfernandez]

I've taken the liberty to add pypa/pip#7102 to the rollout.

[di]

Support in pip has been merged in pypa/pip#7102.

[xavfernandez]

FYI pip 19.3 has been released with manylinux2014 support (& get-pip.py should soon follow).

[hrw]

multibuild uses Travis CI to do builds. Some time ago Travis CI got support for doing AArch64 builds.

Can you add aarch64 builds with manylinux2014 so users will not have to compile wheels on their systems?

[di]

@hrw manylinux2014 adds support for aarch64, but it will be the responsibility of individual package maintainers to add manylinux2014 wheels for their projects once all the tooling is rolled out as described in this issue.

[hrw]

@di I wonder how much work it would be to move from "send us source+wheels and we publish" to "send us source, we build wheels and publish".

Now it can take 45 minutes to do "pip install scikit-learn" on aarch64 compared to few seconds on x86-64 machine. Not counting time needed to find out which development packages to install first to get those Python packages built.

[di]

@hrw A lot. PyPI currently does zero introspection or execution of any user-supplied code. However, there are other repositories (piwheels.org, conda-forge.org) that do something similar, so it's not unprecedented.

However I think this is off-topic for this issue, a better place to discuss further would be in the PyPI issue tracker.

[matthew-brett]

@hrw - for some wheels, it's not very difficult - if they build useful wheels with the default recipe (python setup.py install equivalent). For a lot of scientific Python wheels, there are other libraries that need building, options that need considering, and so on, and the package author needs to resolve.

[mayeut]

auditwheel 3.0.0.0rc1 has been released.
The docker image PR has been updated with this.
The README still needs to be updated. Seems it's the last step for the build environment PR unless other reviewers want to give some feedback.
Before it can be merged, the corresponding quay.io repos have to be created (@takluyver ?)
Shall it be merged in master (and manylinux2010 moved in a branch) or shall it be merged in a branch keeping manylinux2010 the master until CentOS 6 EOL ? I'd go for the latter but any input is welcome.

[takluyver]

I've just created manylinux2014 repositories for x86_64, i686 and aarch64:
https://quay.io/organization/pypa

Let me know if there's anything I've missed. I'm also happy to add you to the quay.io organisation - are you mayeut on quay.io as well?

[mayeut]

I'm also happy to add you to the quay.io organisation - are you mayeut on quay.io as well?

@takluyver, yes, I'm mayeut on quay.io as well, thanks !

[takluyver]

OK, I've invited you. If you create any more repositories, remember to give the bot account write permission for them - you can copy the details from any existing ones.

[mayeut]

First draft for README pushed to #348
If there are no further reviews coming, I'll merge the PR in a manylinux2014 branch on Monday to get things going forward.

[mayeut]

#348 has been merged in branch manylinux2014, first images now deployed in quay.io

[mayeut]

Twine can upload manylinux2014 wheels: https://test.pypi.org/project/pybase64/1.0.1/#files

[mattip]

Is there a central location where all the supported pypa/manylinux* images are catalogued? The README here does not list them, and searching on quay.io does not seem to return anything for manylinux2014

[mayeut]

@mattip, @jayfurmanek, images for manylinux2014 are catalogued in the README of the manylinux2014 branch https://github.com/pypa/manylinux/tree/manylinux2014
I thought I added a link from the master README (manylinux2010) but I didn't. Will do.
You can also see all images on quay.io https://quay.io/organization/pypa

[dwighthubbard]

Now that there is a manylinux2014 container the screwdrivercd project and the python-screwdrivercd packages used by the python_sdv4_templates screwdrivercd python templates have been updated to allow generating manylinux2010 and manylinux2014 packages and publish them using twine using the new containers.

We have updated the redislite package to publish both formats. I'd be interested in knowing if we're missing anything and if/when it might be appropriate to start promoting the manylinux2014 support.

[hrw]

@dwighthubbard next step will be building non-x86 wheels I hope.

[mattip]

I see you are not publishing a manylinux1 wheel. Are you getting complaints from people with old pip versions who do pip install (without --only-binary) and are surprised to get a sdist source tarball that they have to compile?

[ericzolf]

Perhaps your top level README.rst file should already reference https://www.python.org/dev/peps/pep-0599 ? I got confused by manylinux2014 being listed but not referenced via a PEP.

[The-Compiler]

Is there a way to force older pip versions to install manylinux2014 wheels? I thought --platform manylinux2014 would help, but that errors out with ERROR: Can not use any platform or abi specific options unless installing via '--target'.

[The-Compiler]

Oh, another data point:

• virtualenv v16.7.6 (2019-10-16) bundles pip 19.3 (thus works with manylinux2014)
• However, Python's venv/ensurepip bundle pip 19.2.3 as of Python 3.8.1, thus fail to install manylinux2014 wheels.

There seems to be a PR against cpython to update the bundled versions here: python/cpython#16782

[di]

Now that pip==19.3 and auditwheel==3.0.0 have been published, and the PyPUG platform compatibility page has been updated, everything in the "Essential client support", "Enable publication" and "Management of transition" sections has been completed.

Thanks to @brainwane, @mayeut, @lkollar, @pradyunsg, @takluyver, @zachW, and especially @pfmoore for everyone's work here (as well as anyone I may be forgetting).

[brainwane]

• Reword the "not yet widely supported" caveat from the platform compatibility tagging page on PyPUG

@di I still see this on the platform compat tagging page - am I missing something?

[di]

@brainwane In pypa/packaging.python.org#656 I reworded the caveat to say "manylinux2014" instead of "manylinux2010", is that not what you intended?

[brainwane]

Whoops - sorry, I see that now, sorry @di .

Also, I see dockcross/manylinux2014-x64 , dockcross/manylinux2010-x64, dockcross/manylinux2010-x86, dockcross/manylinux1-x64, and dockcross/manylinux1-x86 in the dockcross README. So can we check that off?

[brainwane]

@di could we take a moment sometime this week to refresh this issue, see whether anything in the checklist is finished or obsolete, and see what (if anything) is still waiting to close this?

[di]

I think we can call this finished! Thanks again everyone for your contributions here.