Build python wheels via scikit-build-core

[cary-ilm]

This converts the setuptools configuration for building python wheels to use scikit-build-core, which has better support for CMake. There is no more setup.py; the configuration is entirely in pyproject.toml and the compile/link is done exclusively via cmake.

The build/publish policy is:

• A PR that changes any of the python wheel source/configuration
  (src/wrappers/python/* or .github/workflows/python-wheels.yml)
  triggers a build as a check.

• PRs that change other library source do not trigger a build of the
  python wheels. Note that the primary CI workflow does build and test
  the bindings, although only for a single python version on a single
  arch for Linux/macOS/Windows. The wheel building validates multiple
  python versions and architectures, but involves signifant
  computation/time. Currently, the python wheels are a thin wrapper
  about basic read/write functions that don't add significant
  additional functionality to the library. Any potential problem will
  almost certainly get caught by the primary CI.

• A tag of the form v3.[0-9]+.[0-9]+-rc* (e.g. v3.2.4-rc) triggers
  a full build of the wheels followed by a publish to
  test.pypi.org. This validates release candidates.

• Publishing a release triggers a full build of the wheels followed by
  a publish to pypi.org.

Other changes:

• Consolidate CMake BUILD/INSTALL options in a central place in cmake/OpenEXRSetup.cmake
  and make the pairing of BUILD and INSTALL consistent. The options are:

  option(OPENEXR_BUILD_LIBS "Enables building of main libraries" ON)
  option(OPENEXR_BUILD_TOOLS "Enables building of utility programs" ON)
  option(OPENEXR_INSTALL_TOOLS "Install OpenEXR tools" ON)
  option(OPENEXR_BUILD_EXAMPLES "Build OpenEXR examples" ON)
  option(OPENEXR_INSTALL_EXAMPLES "Install OpenEXR examples" ON)
  option(OPENEXR_BUILD_PYTHON "Build python bindings" OFF)
  option(OPENEXR_TEST_LIBRARIES "Run library tests" ON)
  option(OPENEXR_TEST_TOOLS "Run tool tests" ON)
  option(OPENEXR_TEST_PYTHON "Run python binding tests" ON)

• The wheel build needs the software version, and rather than manual parsing of the openexr_version.h header, it gets it via an invocation of cmake. The new BUILD_LIBS option tells cmake to ignore the libraries entirely in this case (otherwise it would fetch and download libdeflate and Imath).

• The OPENEXR_TEST_LIBRARIES option disables the library tests, which isn't necessary for the python tests during wheel building.

Signed-off-by: Cary Phillips [email protected]

[JeanChristopheMorinPerso]

This looks great!

I left some notes to show how you can fix the problem where you can't import OpenEXR after pip installing it.

I also noticed that some things should be pruned from the wheel:

include/Imath
include/OpenEXR
lib/cmake
lib/libIex-3_3.a
lib/libIlmThread-3_3.a
lib/libImath-3_2.a
lib/libOpenEXR-3_3.a
lib/libOpenEXRCore-3_3.a
lib/libOpenEXRUtil-3_3.a
lib/pkgconfig

These would either pollute site-packages or outright be installed in "random" places.

Little trick, a wheel is just a zip file. So after building it locally, you can inspect it with unzip -l <path.whl>.

I expect the final wheel to only contain these files:

Imath.py
OpenEXR.so
openexr-3.3.0.dist-info/METADATA
openexr-3.3.0.dist-info/WHEEL
openexr-3.3.0.dist-info/entry_points.txt
openexr-3.3.0.dist-info/licenses/LICENSE.md
openexr-3.3.0.dist-info/RECORD

Let me know if you have questions!

[cary-ilm]

Thanks for the guidance and the fixes! For the files you mentioned above that should be pruned, what's the mechanism to do that?

[JeanChristopheMorinPerso]

Thanks for the guidance and the fixes! For the files you mentioned above that should be pruned, what's the mechanism to do that?

I tried setting cmake.targets = ["PyOpenEXR"] thinking that only the PyOpenEXR target would be installed, but it's also installing the includes and static libs. So I guess putting the installation of includes, libs and cmake files in an if block or something? Or maybe there is a cleaner way to tell cmake to not install these when cmake --install <build dir> --targets PyOpenEXR is run...

Maybe you could use CMAKE_SKIP_INSTALL_ALL_DEPENDENCY like

if(SKBUILD)
  set(CMAKE_SKIP_INSTALL_ALL_DEPENDENCY true)
endif()

in the main CMakeLists.txt or in src/wrappers/python/CMakeLists.txt and set cmake.targets = ["PyOpenEXR"] in pyproject.toml?

[JeanChristopheMorinPerso]

I just did a quick test, and setting cmake.targets = ['PyOpenEXR'] and adding OPENEXR_INSTALL = 'OFF' in tool.scikit-build.cmake.define gets us close. We are left with the Imath stuff. CMAKE_SKIP_INSTALL_ALL_DEPENDENCY doesn't seem to have the desired effect...

I'm not sure how we could get rid of the Imath stuff (cmake configs, includes, pkgconfig and lib). But I'm guessing you'll probably know how to do that.

[cary-ilm]

Imath needs an IMATH_INSTALL option so that OpenEXR can propagate OPENEXR_INSTALL to it: AcademySoftwareFoundation/Imath#366

With that change, the contents of the wheel looks good:

./Imath.py
./OpenEXR.so
./openexr.libs
./openexr-3.3.0.dist-info
./openexr-3.3.0.dist-info/RECORD
./openexr-3.3.0.dist-info/licenses
./openexr-3.3.0.dist-info/licenses/LICENSE.md
./openexr-3.3.0.dist-info/WHEEL
./openexr-3.3.0.dist-info/entry_points.txt
./openexr-3.3.0.dist-info/METADATA

[JeanChristopheMorinPerso]

Great! What's is ./openexr.libs in the wheel?

[cary-ilm]

It's an empty directory. I'm not sure where it's coming from.

[JeanChristopheMorinPerso]

Ok thanks. I guess we'll have to start a hunt to find out where it's coming from.

[JeanChristopheMorinPerso]

Umm, openexr.libs is not in thw eheel when I build locally with your latest changes:

Archive:  dist/openexr-3.3.0-cp311-cp311-manylinux_2_38_x86_64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
     9236  2024-02-09 01:08   Imath.py
  3860984  2024-02-11 18:18   OpenEXR.so
     6815  2024-02-11 18:18   openexr-3.3.0.dist-info/METADATA
      117  2024-02-11 18:18   openexr-3.3.0.dist-info/WHEEL
        0  2024-02-11 18:18   openexr-3.3.0.dist-info/entry_points.txt
     1501  2024-02-11 18:18   openexr-3.3.0.dist-info/licenses/LICENSE.md
      536  2024-02-11 18:18   openexr-3.3.0.dist-info/RECORD
---------                     -------
  3879189                     7 files

I'll wait for the workflows to finish before looking further.

[JeanChristopheMorinPerso]

Ah, it's there in the CI generated wheels. I think I know what it is. cibuildwheel runs auditwheel (auditwheel repair) to make sure that the wheel will really work and it's auditwheel that creates this folder. See https://github.com/pypa/auditwheel/blob/db7962149a3e971eba3974a58edcb2226c787f8d/src/auditwheel/main_repair.py#L54-L62.

So it's all good.

[cary-ilm]

@JeanChristopheMorinPerso, I added the sdist step but the wheels don't see to have any additional sourced components. I download an unzip the unbunt-latest artifact, shouldn't it include source? Did I miss a step?

[cary-ilm]

This looks like it's in good shape now, I'm going to merge it and address any further issues separately. @JeanChristopheMorinPerso, thanks for your copious help!