[request] support --disable-gil builds (PEP703)

[tonybaloney]

Description

Please can you add support for building wheels for the --disable-gil version of CPython 3.13.

Similar to debug builds (d), --disable-gil wheels will have an extra suffix (t). The ABI is incompatible with a regular build of CPython 3.13. PyObject has a different data structure.

See pypa/packaging#728 for a related change

Build log

No response

CI config

No response

[joerick]

Cheers Anthony, I was wondering when the need for this was gonna crop up. Given how crucial (and unknown) the C extension ecosystem is to the free-threaded project, I think it makes sense to try to add support for this to cibuildwheel ASAP, so packagers can start trying to build wheels.

I expect the easiest platform to get this working will be Linux, as manylinux already build Python themselves. I'll raise an issue there to see what they think. For the other platforms, the question is how we can get hold of a free-threaded binary build. I think that cibuildwheel building it every time is gonna be too large a cost, so we'd want to get a Mac/Windows build of free-threaded Python from somewhere. Ideally, we start to see them distributed with the alphas.

@zooba, I think you're close to the Python Windows packaging? What's your feeling about CPython distributing a built binary of free-threaded Python, alongside the normal release, in the alpha phase?

[zooba]

I think you're close to the Python Windows packaging? What's your feeling about CPython distributing a built binary of free-threaded Python, alongside the normal release, in the alpha phase?

I am, and obviously we'll do it, but I'm not dropping other work to make it happen myself. There are still big open questions to resolve (e.g. the name, the wheel tag, etc.) and I haven't seen those answered yet (though I'm getting to this post before reading the last week of discuss.python.org, so maybe they have been...)

It wouldn't surprise me if there was a nuget.org release or possibly just a zip file release before an installer-based one, just because of the much greater complexity in handling side-by-side installs of two packages of the same version (which we resolve by changing the package "name", but that has very significant implications for compatibility in MSI bundles, and getting it wrong would very likely leave one or another install irreparably broken). But I think you'd prefer that kind of package anyway? The CI systems are probably set up to use the MSI though, so until we get that sorted out, I wouldn't expect them to preinstall anything else.

[joerick]

Thanks for the response @zooba, that all makes sense to me. You're absolutely right that the nuget package is the one that we'd be most interested in, as that's how we install all the other cpythons, so a free-threaded build on nuget would be ideal for us.

[pradyunsg]

Can cibuildwheel add support for using beta pip releases via an opt-in? That would unblock this during the CPython beta phase and would also align with pip's goals in this area.

[henryiii]

It wouldn’t help I think, we need pypa/manylinux#1564 and that’s blocked awaiting on the pip release. (We also need macOS binaries for free threading that are planned for beta 2)

When is the release planned? I was hoping it is any day.

One option would be to only use the beta version of pip on 3.13. But that would still need to happen in manylinux.

[henryiii]

So, while waiting for binaries, how are we going to support this? A new identifier? Opt in or opt-out?

[webknjaz]

Would #1718 help with this?

[colesbury]

It looks like the manylinux images are out. There's no official macOS build yet, but it'd be great if cibuildwheel can start supporting Linux and Windows free-threaded builds as there are a large number of projects that depend on cibuildwheel and this can help unblock them.

For identifiers, I think it would makes sense to follow the platform identifier tags. For example: cp313t-manylinux_x86_64 (note the "t"). What does opt-in or opt-out mean in this context? Is there a default value for CIBW_BUILD?

[henryiii]

No, we don't have a default, but we do have an opt-in mechanism for beta versions of Python. A similar opt-in mechanism could be added here, but it probably should be also supported in TOML, unlike the Python beta option. Or we could just require people skip the free-threaded build if they don't support it, but that would mean cibuildwheel is a lot less likely to work out of the box.

We could kind of punt on this for a while, though, as it will be protected by the pre-release flag for now.

[mayeut]

A similar opt-in mechanism could be added here, but it probably should be also supported in TOML.
We could kind of punt on this for a while, though, as it will be protected by the pre-release flag for now.

To get things started, I used the pre-release flag for now in #1831 to see the minimum required set of changes needed to get everything to work.

I don't know how many users do use the pre-release flag but, while they probably do accept that using a pre-release might break things, this feels like it could break much more users than usual. I do have a simple project that never failed using the pre-release flag that now fails to test properly with the nogil build (warning: The global interpreter lock (GIL) has been enabled to load module 'pybase64._pybase64', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0).

I will add another commit in #1831 to propose an opt-in mechanism.

Edit: The opt-in mechanism has been added and the PR is now ready for review.

[henryiii]

If most work but just produce a warning, not sure if we want to encourage opting out. Building a GIL enabled binary for python3.13t is much better than no binary and helps test this out. It’s harder to remove a toml option. But we could flip the default in the future, so okay with it. I’d like to try a couple of more complex packages to see. Feedback welcome from projects, too.

[mayeut]

On the one hand, I'd like to encourage testing this option, on the other:

• It's still marked as an experimental feature
• It will break some updates of cibuildwheel (and thus might slow down generic cpython 3.13 adoption)
• Projects using abi3 might have more troubles:
  • they might just not build (that was the case for psutil)
  • pre-release might always be enabled and thus, they end up publishing cp313t wheels during the non stable ABI phase which might cause more problems than solutions if the ABI does break during the BETA phase.

So, maybe a middle ground would be to get a first version out with the feature disabled as a default but add a prominent warning in the release note that it will be enabled by default in the following version. This would at least give some projects the time to prepare between versions of cibuildwheel rather than just react to the change.

Feedback welcome from projects, too.

Indeed !

On that note, I did not find much guidelines to help packagers. It's most likely a bit too soon to have those but adding links in the docs might be worth it. The only thing in there for now is https://docs.python.org/3.13/whatsnew/3.13.html#free-threaded-cpython

[rgommers]

Since it's not recommended to use free-threaded builds in production yet and many projects won't build or build and be somewhat broken at runtime, opt-in seems better for now. Most packages also won't work out of the box, if only because there are no wheels for their dependencies yet.

For scientific Python projects we plan to start producing wheels when cibuildwheel support lands, and upload them to the shared index for nightlies (https://anaconda.org/scientific-python-nightly-wheels/). Other subsets of the community will hopefully start similar testing efforts. A change from opt-in to opt-out should probably come after at least a few months of observing that such wheels are largely working and stable?

[henryiii]

A guide is being worked on: Quansight-Labs/free-threaded-compatibility#15

[henryiii]

Released (Linux and Windows).

[hugovk]

On that note, I did not find much guidelines to help packagers. It's most likely a bit too soon to have those but adding links in the docs might be worth it. The only thing in there for now is docs.python.org/3.13/whatsnew/3.13.html#free-threaded-cpython

Yep, we don't have it yet in the CPython docs, but a guide for C API extension authors is planned in python/cpython#119241.

[henryiii]

Also see Quansight-Labs/free-threaded-compatibility#15.

[ngoldbaum]

Over at numpy/numpy#26512 I'm trying to set up free-threaded wheels. According to the release notes, windows should be supported, but e.g. on this build it can't seem to find the shared library:

Program python found: YES (C:\Users\runneradmin\AppData\Local\Temp\build-env-_csx0kis\Scripts\python.exe)
  Could not find Python3 library 'C:\\Users\\runneradmin\\AppData\\Local\\pypa\\cibuildwheel\\Cache\\nuget-cpython\\python-freethreaded.3.13.0-b1\\tools\\libs\\python313.lib'
  Run-time dependency python found: NO (tried sysconfig)

On the cp313 build from the same run, it gets past that step:

Program python found: YES (C:\Users\runneradmin\AppData\Local\Temp\build-env-h1qfb9dq\Scripts\python.exe)
  Run-time dependency python found: YES 3.13
  Has header "Python.h" with dependency python: YES 

Any idea what's going wrong?

[mayeut]

The library is named python313t.lib for the free-threaded variant.
There might be something missing in meson for this ?

[ngoldbaum]

Makes sense, I doubt anyone has tried to build on windows yet.

[henryiii]

We now support all free-threaded options in 2.19!