Python 3.11

[dopplershift]

Are there any known issues standing in the way of getting wheels for Python 3.11 up?

If not, would it be helpful if I ported @ocefpaf's work adding cibuildwheel to cftime over to this repo?

[jswhit]

No issues for 3.11 that I know of. @ocefpaf has a netcdf-4python cibuildwheel repo for windows wheels - but we're still using the macpython site for macosx and linux wheels.

[ocefpaf]

It would be nice to port a cibuildwheel for Linux, macOS, and, if we can Windows to this repo like cftime. Note that what I did for Windows is not cibuildwheel. It is a trick to use conda-forge dependencies for netcdf-c and hdf5 instead of building them ourselves in CIs and producing a valid/compliant wheel that is pip instalable.

I believe that porting macOS and Linux to ocibuildwheel will be relatively easy but I have no idea how to start a build of hdf5 and netcdf-c for Windows on CIs. This is how h5py does it BTW. I like that it is all Python but they are still using AppVeyor for it :-/

[jswhit]

I've created 3.11 wheels for 1.6.1 at https://github.com/MacPython/netcdf4-python-wheels/releases/tag/v1.6.1rel. @ocefpaf could you regenerate the 1.6.1 release at https://github.com/ocefpaf/netcdf4-win-wheels/releases/tag/v1.6.1 with 3.11 wheels? I can transfer the wheels to pypi once they are available.

[jswhit]

@ocefpaf why not used the conda-forge dependency trick for linux and macosx also?

[ocefpaf]

@ocefpaf why not used the conda-forge dependency trick for linux and macosx also?

Possible but we should discuss the pros and cons there:

• Pros: curated set of dependencies with broader community, easier to setup, probably more up-to-date dependencies
• Cons: the setup won't have a lot of freedom to use a different version of deps that are not in conda-forge or with different build flags/options, wheels may be a bit larger due to extra options added in conda-forge. (When building for wheels authors tend to use bare bones options.)

@ocefpaf could you regenerate the 1.6.1 release at https://github.com/ocefpaf/netcdf4-win-wheels/releases/tag/v1.6.1 with 3.11 wheels?

Doing that now.

[ocefpaf]

@jswhit The Python 3.11 wheel for Windows can be downloaded here:

https://github.com/ocefpaf/netcdf4-win-wheels/suites/9117621053/artifacts/423175481

If you want to inspect the full run and CI you can check https://github.com/ocefpaf/netcdf4-win-wheels/actions/runs/3388771868 and ocefpaf/netcdf4-win-wheels#5

PS: just to get back to your question above. To be honest I'm on the fence there. As a conda-forge packager I'd love to see more wheels using our packages as source for their non-Python dependencies. However, at least for wheels that are "easy enough to build" I believe that cibuildwheel is the community standard, and the best solution for 95% of the cases. I guess that the question is: Is netcdf4-python in the 5%? We will only know the answer if we try to send a PR to implement cibuildwheel here and see how difficult that will be. @dopplershift are you planning sending that PR? I can try but I cannot promise anything. (End of school year is near and my 2.5 year old will be home all the time, my PRs will decrease in number and quality a lot in the upcoming days.)

[jswhit]

Python 3.11 wheels uploaded to pypi. I think netcdf4-python may be in that 5% of packages with so many C lib dependencies that using the conda-forge libs in the wheels makes sense.

[dopplershift]

@ocefpaf I'm happy to put doing the PR on the list, but it's probably not happening for a couple weeks (it's a little less of a priority with the 3.11 wheels up) since I urgently need to get a MetPy release out the door. But otherwise, if no-one has gotten to it by then I'm happy to take a look and improve my cibuildwheel skills.

[ocefpaf]

Python 3.11 wheels uploaded to pypi. I think netcdf4-python may be in that 5% of packages with so many C lib dependencies that using the conda-forge libs in the wheels makes sense.

I made an attempt here: ocefpaf/netcdf4-win-wheels#6

There are a few things we need to solve before going that route and I'd like to try cibuildwheels too before deciding.

1. We have 3 failing archs ppc, aarch, and osx_arm64. At the least the last one should be fixed due to its popularity. The first one may require some bugfixes. See Seeing test failures on conda-forge building for ppc64le #1196
2. We need to ship the same compression options that are present in the current wheel build system. That will require us to solve Add SZip support using libaec conda-forge/hdf5-feedstock#179 (comment)
3. Conda-forge's minimum numpy is not 100% equivalent to oldest-supported-numpy, we diverge in some archs. That is probably minor but we need to investigate if this won't create any breakages from what users are expecting.

One advantage of this approachis that conda-forge's curl/openssl are updated more frequently and considered "safer." Going this route would also solve #1179.

[jswhit]

that's a great start @ocefpaf, thanks!

[dopplershift]

Looks like we're back to this for 3.12 now.

[ocefpaf]

While we are still hitting some issues with the https hosted files, I believe we can close this one. Wheels are all created via cibuildwheel and using GitHub Actions. I could use more eyes in the docker image used for the Linux build though. See https://github.com/ocefpaf/netcdf-manylinux