3.11 builds failing with v2.8.1 on manylinux2010 images

[godlygeek]

Description

With:

        cibw_python: [ "cp37-*", "cp38-*", "cp39-*", "cp310-*", "cp311-*"]

and

        env:
          CIBW_PRERELEASE_PYTHONS: True

I'm seeing this error:

Setting up build environment...
  
      + /opt/python/cp38-cp38/bin/python -c 'import sys, json, os; json.dump(os.environ.copy(), sys.stdout)'
      + which python
  cibuildwheel: python available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert python above it.
  Error: Process completed with exit code 1.

This didn't occur with 2.8.0 - I suspect it's somehow related to this changelog entry:

🛠 The GitHub Action will ensure a compatible version of Python is installed on the runner (#1114)

I'm also seeing

Run actions/setup-python@v4
  with:
    python-version: 3.7 - 3.10
    update-environment: false
    token: ***
  env:
    CIBW_BUILD: cp311-*
    CIBW_ARCHS_LINUX: x86_64
    CIBW_MANYLINUX_X86_64_IMAGE: manylinux[20](https://github.com/bloomberg/memray/runs/7420878603?check_suite_focus=true#step:3:21)10
    CIBW_MANYLINUX_I686_IMAGE: manylinux2010
    CIBW_PRERELEASE_PYTHONS: true
    CIBW_TEST_EXTRAS: test
    CIBW_TEST_COMMAND: pytest {package}/tests

in the build log - perhaps that "python-version: 3.7 - 3.10" is the problem?

Build log

https://github.com/bloomberg/memray/runs/7421778954?check_suite_focus=true

CI config

https://github.com/bloomberg/memray/blob/f9bfaacea95c9c565189da2eeb121a3303c11c71/.github/workflows/build_wheels.yml

[henryiii]

That’s not what that range means. It uses one version from that, and the outer host setting shouldn’t matter. I wish we printed the details of the mismatch. We are using 3.11b4 now. Don’t see how that would change this either, though. Am traveling and can’t check quite yet.

[henryiii]

Why is it using /opt/python/cp38-cp38/bin/python? Host is 3.10 and target is 3.11. Don’t get where 3.8 is sneaking in.

[godlygeek]

I'm past my depth in debugging this, but if there's anything I can run to help you make sense of it, I'd be happy to. I've added a type -a python to before-all and it prints out:

  python is /opt/python/cp38-cp38/bin/python
  python is /usr/bin/python

but I'm not sure where the /opt/python/cp38-cp38/bin/python is coming from...

Likewise, printing $PATH is showing me

  PATH=/opt/python/cp38-cp38/bin:/opt/rh/devtoolset-8/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

This could be something I've got misconfigured, but v2.8.1 is failing in a way that v2.8.0 didn't...

[mayeut]

manylinux2010 does not have CPython 3.11
support for this image will be dropped before ABI stability of CPython 3.11 thus the beta has been dropped on this image to prevent building 3.11 with a Python that will never be ABI stable on that image.
You'll have to use manylinux2014 image to build CPython 3.11

• Drop support for manylinux2010 images August 1st, 2022 manylinux#1281
• pypa/manylinux@97c4ab0

[godlygeek]

Oof. OK... Well, that's not good news for me, since I do still need RHEL 6 compatibility, but at least that answers what changed.

Thanks for helping me look into this.

[joerick]

Thanks for the debugging. We could use a better error message for this then. @mayeut I'm guessing manylinux1 also won't get Python 3.11?

[henryiii]

We need to detect this then, rather than grabbing a random Python. :)

manylinux1 is “dead”, it’s just still being built while CI doesn’t break, so it’s not getting anything new. Not sure you can build 3.11 on CentOS 5.

You can use an override to build 3.11 with manylinux2014. In fact, IIRC, NumPy switched to 2014 for 3.10, since anyone on 3.10 should be on a 2014 compatible platform. Doesn’t help to have higher compatibility than your dependencies.

[mayeut]

I'm guessing manylinux1 also won't get Python 3.11?

Correct. The manylinux1 image does not even have Python 3.10

We need to detect this then, rather than grabbing a random Python. :)

It's correctly detected. The error message "cibuildwheel: python available on PATH doesn't match our installed instance. If you have modified PATH, ensure that you don't overwrite cibuildwheel's entry or insert python above it." relates to that (could be clearer by checking the path added to PATH exists in the first place).

The /opt/python/cp38-cp38/bin/python python mentioned is the one used as a build driver inside the container (counterpart to the host python running cibuildwheel).

[joerick]

Yep, it is detected, it's just the error message is confusing because that message is assuming that the user has fiddled with PATH to break it, not that it never existed in the image.

We can probably add a different message, like if not python_bin.exists(): raise Exception(f'Python {python_configuration.version} doesn't exist in image {image}'). And even perhaps a fail-fast before the builds start.

[joerick]

The /opt/python/cp38-cp38/bin/python python mentioned is the one used as a build driver inside the container (counterpart to the host python running cibuildwheel).

I don't recall adding this to PATH in cibuildwheel. Perhaps it's added in manylinux?

[mayeut]

Perhaps it's added in manylinux?

The only python in the PATH in manylinux images is the system one (when it exists, on some images the system python is python3)

I probably wasn't clear enough about /opt/python/cp38-cp38/bin/python.

The /opt/python/cp38-cp38/bin/python python mentioned is the one used as a build driver inside the container (counterpart to the host python running cibuildwheel).

It's almost always used by cibuildwheel with the full file path:

cibuildwheel/cibuildwheel/oci_container.py

Line 46 in 7c7dda7

UTILITY_PYTHON = "/opt/python/cp38-cp38/bin/python"

except for before_all step:

cibuildwheel/cibuildwheel/linux.py

Lines 128 to 137 in 7c7dda7

	if before_all_options.before_all:
	log.step("Running before_all...")
	env = container.get_environment()
	env["PATH"] = f'/opt/python/cp38-cp38/bin:{env["PATH"]}'
	env["PIP_DISABLE_PIP_VERSION_CHECK"] = "1"
	env = before_all_options.environment.as_dictionary(
	env, executor=container.environment_executor
	)

[joerick]

ahh.. gotcha. So the cp38 in PATH thing really is a red herring. It's only a result of trying to investigate this using before-all.