Update default image away from manylinux2014

[dralley]

Description

CentOS 7 (on which manylinux2014 is based) goes end-of-life in only a few months (June) - as such it's probably a good idea to look for a new default base image. Almost certainly manylinux_2_28.

Build log

No response

CI config

No response

[dralley]

And also, although this is a separate issue, decide on a timeline for dropping support for manylinux2014 from other ecosystem tools (probably not soon, but still, dates should be decided and communicated in advance).

[henryiii]

FYI, manylinux1 and manylinux2010 were supported well past CentOS EoL, in fact manylinux1 is still receiving automated updates and is still supported in cibuildwheel (CentOS 5). 20% of systems cannot support manylinux_2_28 (https://mayeut.github.io/manylinux-timeline/), based solely on the GLIBC version (you also need a recent pip version, which might limit it further). 70% of projects still distribute manylinux2014 wheels.

So while this might change, I don't think it's likely to until after the EoL deadline. (That might help it start to change).

[EwoutH]

We could also do it conditionally, based on Python version. So for example, practically all Python 3.12 consumers support glibc 2.31 or higher. Almost all Python 3.11 consumers support 2.28 or higher.

So you could end up with something like:

• Python 3.6-3.9: Default to manylinux_2_17 (== manylinux2014)
• Python 3.10: Default to manylinux_2_26
• Python 3.11: Default to manylinux_2_28
• Python 3.12: Default to manylinux_2_31

Of course this needs to be properly announced, and maybe some policy for instated to update those numbers.

[joerick]

Hmm..! Nice idea, downside would be that it is a bit wasteful in time and bandwidth, as a default. Much of the time of a build is downloading the container, plus, some users do heavy stuff in before-all, which would be done many times for such a setup. Plus, most users don't really need the newer standards - manylinux2014/2010/1 are by far the most popular options today. https://sourcegraph.com/search?q=context:global+%28CIBW_%29%3FMANYLINUX%5B_-%5DX86_64%5B-_%5DIMAGE&patternType=regexp&sm=0

[joerick]

a great reference here is @mayeut's manylinux timeline

[mayeut]

As @joerick said, it's likely just a waste of resources to change the default conditionally on the python version and, users can already do that if they want(/need) to using overrides.

With the current stats for non-EOL python versions, I'd say a good time to change the default could be after AmazonLinux 2 EOL (2025-06-30, I think that's most of GLIBC 2.26 downloads) or Python 3.9 EOL (2025-10-31).

[EwoutH]

I think we can already safely go to 2.26 as the new default.

The fact that 2.17 (==2014) is so much used, is not because target systems don't support better, it's simply because it's the current default.

[mayeut]

I think we can already safely go to 2.26 as the new default.

There are no images for 2.26. The next supported image is 2.28 hence the proposition in #1772 (comment)

[EwoutH]

Support has risen to 92.8% according to the manylinux timeline.

According to glibc versions the distro that don't support glibc 2.28 is mainly Amazon Linux 2, which I think they can handle (and a newer Amazon Linux 2023 image has been available for a while).

Especially with CentOS 7 now being end of life, it might be time to update the default to manylinux_2_28.

Edit: You can also see:

• the issues/PRs tracking this issues rising sharply
• the new issues in this repo rising:
  
• the issues in the manylinux repo rising:
  

[EwoutH]

I opened a PR to move the default: #1988.

[henryiii]

The last statement on the matter was:

I'd say a good time to change the default could be after AmazonLinux 2 EOL (2025-06-30, I think that's most of GLIBC 2.26 downloads) or Python 3.9 EOL (2025-10-31).

I think it would be good to get agreement or at least one maintainer on board before just going ahead and making the change.

Another option, by the way, is to make the default arch-specific, and "special" archs could default to 2_28, as 2014 doesn't work as well for special archs.

Moving away from 2014 will be quite disruptive as it has to be done on lower level dependencies first. For example, if numpy were to go 2_28, then every package using it to build would start either building numpy from scratch, taking much longer, or would start failing if it's not buildable. Maybe making this a recommendation for a while, or setting a change-over date, etc. would be a better approach?

Also, previous versions of manylinux have gone 4+ years past EoL of the distro they were based on.

[EwoutH]

Thanks for your insights. I also openend the PR to get some insights in if there are any complications implementation wise (such as the i868 issue) and to get the discussion started again.

Maybe making this a recommendation for a while, or setting a change-over date, etc. would be a better approach?

I like the idea, but I'm curious what would your exact recommendation be? Asking repos to set specific manylinux versions, instead of pointing to the default one? That would make the point of a "default" manylinux version a bit mood right?

Another idea I have right now is to add a flag "USE_FUTURE_DEFAULT". By default it's False, keeping the current behavior, but flipping it to True will allow you to opt-in already to the future default, or set your version manually to the current default (in this case manylinux2014).

First we could just add it, then we could update it to throw a warning that's soon changing, and then we can flip the default value.

Another option is that in theory we could also build both manylinux2014 and manylinux_2_28 for some time. But I don't know what the exact implications of that would be.

Anyway, this will hurt in some way anyway. And currently it's also already hurting repos, by CentOS 7 being EOL (also see the motivation in the PR).

[cdce8p]

Another option is that in theory we could also build both manylinux2014 and manylinux_2_28 for some time. But I don't know what the exact implications of that would be.

A while ago I experimented with using manylinux_2_28 as the default to compile mypy. We switched over after we saw that even with 2_28 the wheels are still 2014 compatible. The repair wheel step changed the filename suffix to manylinux2014_x86_64.manylinux_2_28_x86_64.whl.

Besides that, 2_28 was a nice albeit small performance improvement.

mypyc/mypy_mypyc-wheels#81
https://pypi.org/project/mypy/#files

[EwoutH]

@mayeut @joerick I'm curious what your current views on moving the default manylinux image are.

[henryiii]

Another option is that in theory we could also build both manylinux2014 and manylinux_2_28 for some time. But I don't know what the exact implications of that would be.

Building two wheels by default is out. Packages can fairly easily do that if they want (I did it for manylinux1 and 2010 for a long time for boost-histogram), and we could add docs on it, but we shouldn't double the space and time and drastically complicate the internals to build multiple wheels from a single identifier. This would also run before-all twice, and it would need new ways to customize just one of the two linux builds (since the images are not identical).

Auditwheel will add a lower tag if you aren't using any symbols from a newer GCC, that's why you got the 2014 tag too, I believe. It won't always happen for all projects.

Another idea I have right now is to add a flag "USE_FUTURE_DEFAULT"

I don't like the idea of adding a flag. Someone can always just set these to what they want, and the flag would eventually change meaning for the next bump, etc. An alternative to this would be maybe to push harder for projects to always specify this so they don't depend on our default? That's really quite similar to my previous suggestion of suggesting 2_28 as a recommendation in the docs.

Anyway, this will hurt in some way anyway. And currently it's also already hurting repos, by CentOS 7 being EOL

We've been solving the issues by adjusting mirrors. I expect it will (has?) stabilize. Manylinux1 and 2010 were maintained years after EoL.

[henryiii]

PS: I'm carefully not giving an opinions about the bump date, just pointing out the only opinions I've seen so far are 2025-06-30 and 2025-10-31, etc. I do think it makes sense to set a date and list it in our docs as a change over date. Packages are strongly recommended to pin to minor versions or better, so making a default change is as safe and painless as it reasonably can be.

[EwoutH]

Thanks for your replies and the insight!

One thing I’m thinking of, that if you are using a default most of the time you’re okay bit it being updated once in a while, and you might even expect it. That’s why people use Python 3.x, ubuntu-latest, etc. If you wanted strict control you likely have specified a version yourself.

Like you pointed out, the same goes for pinning the minor cibuildwheel version. For most projects, friendly PRs would be updated.

I’m curious about a few other things. First, do you see an elegant way to throw an info / warning message that we changing the default?

Second, I initially read this wrong:

Python 3.9 EOL (2025-10-31).

I read it as Python 3.8 EOL, which is 2019-10-14. Six weeks seems like a fine timeframe for an announcement and info message.

Finally, from a default I would say it’s broadly supported, not based on a end-of-life host. For me it seems perfectly fine to keep it as a supported option, but it just might be nice to move the default to a version that’s fully supported.

[joerick]

Regarding the exact date, I'm happy to defer to @mayeut who is a lot closer to this part of Python packaging.

His previous statement was:

I'd say a good time to change the default could be after AmazonLinux 2 EOL (2025-06-30, I think that's most of GLIBC 2.26 downloads) or Python 3.9 EOL (2025-10-31).

If there's a toss-up, my bias would be to go for the earlier date of the two. I'm happy for him to decide though.

I do think it makes sense to set a date and list it in our docs as a change over date. Packages are strongly recommended to pin to minor versions or better, so making a default change is as safe and painless as it reasonably can be.

Big fan of this idea. We can also mention it in the next release notes.

[mayeut]

I just created pypa/manylinux#1671 to drop python 3.6 & 3.7 from the manylinux images (mostly in order to add 3.14 without image size growing or build times getting much longer).

3.6 will be dropped for sure. Depending on feedback for 3.7 but given more than 50% of downloads are from AmazonLinux 2 or so it seems, it is reasonable to drop it at that time IMHO.

I think it would make sense to change the default at the same time (which is just over 1 month before AmazonLinux 2 EOL).

[EwoutH]

Thanks for your perspective. It feels a bit weird to keep the default version hosted on end-of-life OSes, because one single cloud OS version isn't compatible. Since it's all cloud usage, they can all relatively easily migrate to the 2023 version of that OS. Or just set manylinux2014 as the default.

So I'm not saying that we shouldn't support manylinux2014 anymore. Just not as default.