Accept macosx_10_9 platform for Big Sur

[memeplex]

Environment

• pip version: 20.2.4
• Python version: 3.8.6
• OS: MacOS 11 (Big Sur)

Description

The update to Big Sur changed the major version of MacOS. AFAICS this is not really related to internals but more because of face lift and commercial concerns. But it's forcing pip to build everything from sources (which is not only slow but tends to fail for complex packages like numpy).

I'm using the --platform flag as a workaround but it requires to also specify --target and --only-binary, which is not really convenient.

Expected behavior

Wheels for previous versions are used.

How to Reproduce

1. pip install numpy
2. Tries to build.
3. Fails to build

Instead:

1. pip install --platform macosx_10_9_x86_64 --only-binary=:all: --target=/tmp numpy
2. Downloads wheel
3. Works fine

Output

This is not really relevant, I'm not concerned with the reasons the build failed (related to blas and lapack) but because the build happens at all.

[memeplex]

I've been using this horrible hack:

    version_str, _, cpu_arch = platform.mac_ver()  # type: ignore
    version_str = "10.15.1"

in site-packages/pip/_vendor/packaging/tags.py and after installing my environments everything seems to work fine.

The solution to this issue may be in changing pip or in building wheels for 11, this depends on the level of compatibility of MacOS 11 wrt 10.

This is OT, but do you know why there aren't wheels for 11 yet and how long does the build usually take?

[pradyunsg]

This is blocked on pypa/packaging#319.

[memeplex]

@MarioCatuogno complex builds may fail because of many reasons and usually fail... But this report is about having to build those packages at all instead of using a pre-built wheel. The cause is clear, pip is not seeing a wheel built for MacOS 10.x as valid for MacOS 11.

[uranusjr]

Marking the comment as Off-topic since it is. @MarioCatuogno the error you’re seeing is a Numpy issue and needs to be reported there. pip cannot do anything about it.

[jaredrcleghorn]

@pradyunsg what do you mean by blocked?

[uranusjr]

In project management terminology, “blocked” means something (“blocking” issue) needs to be done first before the thing at hand (“blocked” issue) can be addressed.

[jaredrcleghorn]

@uranusjr Gotcha, so the problem is in packaging, not pip directly? Or will it need to be addressed in both?

[ogrisel]

For information, it seems that the macos_10_9 wheels of scikit-learn are binary compatible with a mac running the Intel version of Big Sur.

In particular for the Python package shipped as part of the conda-forge distribution, platform.platform() reports macOS-10.16-x86_64-i386-64bit instead of macOS-11.0.1-x86_64-i386-64bit so that pip 20.2.4 running on Intel / Big Sur accepts those macos_10_9 wheels silently and all scikit-learn tests pass.

So from scikit-learn's point of view it seems fine for a future version of pip to accept macos_10_9 wheels for Big Sur. But I don't know for sure if binary compatibility is guaranteed more generally or not.

[uranusjr]

I believe binary compatibility is guaranteed. Not sure if it’s said explicitly, but pypa/packaging#319 is contributed by Apple employees (using their Apple emails), and the implementation essentially declares macOS 11 to be compatible with 10.x.

[fxcoudert]

Binary compatibility for Intel Big Sur is guaranteed with earlier macOS versions

[memeplex]

Please take no offense, but I find hard to believe this is taking too long, I mean:

1. Many, many developers use python in MacOS.
2. About 10 days passed since Big Sur release, and the beta had been out there for some months to test, and the issue was indeed known before the release.
3. The fix seems to be easy, given that packages are compatible with previous version.
4. The problem is serious: at least the scientific stack is very hard to build because of its dependencies (see reports of this issue for numpy, for example, which is one of python most popular packages).

Am I missing something? How is everybody else working around this?

[ogrisel]

This has been fixed in pypa/packaging#319 and related PRs in CPython so I guess the fix will be released soon-ish. It "just" needed someone with the time an knowledge to review the pull requests to add support for Big Sur to CPython related repos. Ironically the person who reviewed that last PR in this case works at Microsoft.

If Apple paid more full time employees to work as core developers on major open source projects such as CPython such fixes would be faster included when they release a new version of their OS.

Pending that, the easy workaround is just to wait for some time before upgrading to a new OS while open source maintainers do the work at their own pace instead.

[memeplex]

Thanks!