[CI][Python] Github Actions macOS runner fails to build PyArrow 13 when Python was built against older macOS.

[DylanLukes]

Describe the bug, including details regarding any error messages, version, and platform.

Minimal Reproduction

A minimal reproduction and run is available here: https://github.com/DylanLukes/pyarrow-ci-macos13-minimal-repro/actions/runs/6606055123/job/17941866516

Issue Description

As PyArrow 13.0.0 has no wheels, I've been having to compile the Cython extensions for CI. The Arrow 13.0.0 header files make use of C++ features which are only available on macOS from 13 onwards, for example:

/usr/local/include/arrow/type.h:1844:29: error: 'get<std::string, arrow::FieldPath, std::string, std::vector<arrow::FieldRef>>' is unavailable: introduced in macOS 10.13
          return IsName() ? &std::get<std::string>(impl_) : NULLPTR;

It turns out that the macos-latest runner is, in fact, not the latest, that would be macos-13. Switching the runner to macos-13 however doesn't fix the issue! What's going on? Inspecting the logs for my own project's failed CI:

Installing project in development mode
  error: subprocess-exited-with-error
  
  × Building wheel for pyarrow (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [468 lines of output]
      <string>:34: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
      WARNING setuptools_scm.pyproject_reading toml section missing 'pyproject.toml does not contain a tool.setuptools_scm section'
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-10.9-universal2-cpython-[31](https://github.com/DylanLukes/renkon/actions/runs/6600268842/job/17930093683#step:9:32)2
      creating build/lib.macosx-10.9-universal2-cpython-312/pyarrow
      copying pyarrow/orc.py -> build/lib.macosx-10.9-universal2-cpython-312/pyarrow
      copying pyarrow/conftest.py -> build/lib.macosx-10.9-universal2-cpython-312/pyarrow

Mac OS X... 10.9?

It's not clear why, but setuptools seems to be inferring the incorrect system version, which in turn propagates down to CMake, which then results in the aforementioned compilation error. Checking the system version in a few ways I get:

> uname -a
Darwin Mac-1697921125444.local 22.6.0 Darwin Kernel Version 22.6.0: Fri Sep 15 13:39:52 PDT 2023; root:xnu-8796.141.3.700.8~1/RELEASE_X86_64 x86_64

> gcc -dumpmachine
x86_64-apple-darwin22.6.0

> python -c "import platform; print(platform.platform())"
macOS-13.6-x86_64-i386-64bit

Workaround

While it doesn't fix setuptools thinking it's building a wheel for Mac OS X 10.9 rather than macOS 13, we can get CMake back on the right page with the following extra step:

...
      - if: runner.os == 'macOS'
        run: echo "MACOSX_DEPLOYMENT_TARGET=$(sw_vers -productVersion)" >> $GITHUB_ENV
...

Diagnostic Progress So Far

I was pretty baffled by this. I'm not sure where setuptools is getting 10.9 from. I've not had any issue like this on a local Mac running macOS 14, it seems specific to the GitHub Actions runner environment?

I then went poking around the details of pip inspect and the contents of the packaged Python the setup step installs. AT first glance it appears to have been built against yet another different version (macOS 11), as it's marked as python-3.12.0-macos11.pkg in the accompanying setup.sh.

But, if we actually run the package (or expand it with pkgutil --expand and take a look inside...), we find that in the accompanying license, readme, and welcome messages it refers to 10.9!

This package will install Python 3.12.0 for macOS 10.9 or later.

Aha! That explains it.

Thoughts

It seems to me that despite this being the case, building pyarrow should work. Yes, 10.9 is not supported... but, we're not on 10.9. And this build can work perfectly well, so long as CMake is not misled about which MACOSX_DEPLOYMENT_TARGET to use.

So, if Arrow 13.0.0 requires macOS 13 in its header files (it won't work on older systems anyways), it seems reasonable that python/setup.py should ensure that rather than deferring to a Python installation which may be misleading it.

Component(s)

C++, Continuous Integration, Python

[raulcd]

Hi Dylan,
Thanks for the great diagnostic. Pyarrow 13.0.0 is not supposed to be Python 3.12 compatible. We are releasing Arrow 14.0.0 and we are providing wheels for Python 3.12, there which I do think should work, see:

• [Python][Packaging] Support Python 3.12 and upload wheels to PyPI #37880

If you want to give those wheels a try you can install them from our nightly builds, take into account that those are not official releases and we are in the process of releasing 14.0.0 which (unless something wrong happens on verification) could be available in the next days.
See for the nightlies wheel installation: https://arrow.apache.org/docs/dev/developers/python.html#installing-nightly-packages

[jorisvandenbossche]

Going to close this issue, as I don't think there is anything actionable left.