Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Python 3.12. #213

Merged
merged 2 commits into from
Sep 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/workflows/conda-cpp-tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -69,19 +69,19 @@ jobs:
pull-request:
# amd64
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '11.8.0', LINUX_VER: 'rockylinux8', GPU: 'v100', DRIVER: 'earliest', DEPENDENCIES: 'oldest' }
- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
# arm64
- { ARCH: 'arm64', PY_VER: '3.10', CUDA_VER: '12.0.1', LINUX_VER: 'ubuntu20.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '12.0.1', LINUX_VER: 'ubuntu20.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
nightly:
# amd64
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '11.4.3', LINUX_VER: 'rockylinux8', GPU: 'v100', DRIVER: 'earliest', DEPENDENCIES: 'oldest' }
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '12.0.1', LINUX_VER: 'rockylinux8', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
# arm64
- { ARCH: 'arm64', PY_VER: '3.10', CUDA_VER: '12.2.2', LINUX_VER: 'ubuntu22.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'oldest' }
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu22.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '12.5.1', LINUX_VER: 'rockylinux8', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.12', CUDA_VER: '12.5.1', LINUX_VER: 'rockylinux8', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
"

TEST_MATRIX=$(yq -n 'env(MATRICES) | .[strenv(BUILD_TYPE)]')
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/conda-python-build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -62,11 +62,15 @@ jobs:
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04' }
- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu22.04' }
- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04' }
- { ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu22.04' }
- { ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04' }
# arm64
- { ARCH: 'arm64', PY_VER: '3.10', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu22.04' }
- { ARCH: 'arm64', PY_VER: '3.10', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04' }
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu22.04' }
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04' }
- { ARCH: 'arm64', PY_VER: '3.12', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu22.04' }
- { ARCH: 'arm64', PY_VER: '3.12', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04' }
"

MATRIX="$(
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/conda-python-tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -72,19 +72,19 @@ jobs:
pull-request:
# amd64
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '11.8.0', LINUX_VER: 'rockylinux8', GPU: 'v100', DRIVER: 'earliest', DEPENDENCIES: 'oldest' }
- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
# arm64
- { ARCH: 'arm64', PY_VER: '3.10', CUDA_VER: '12.0.1', LINUX_VER: 'ubuntu20.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '12.0.1', LINUX_VER: 'ubuntu20.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
nightly:
# amd64
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '11.4.3', LINUX_VER: 'rockylinux8', GPU: 'v100', DRIVER: 'earliest', DEPENDENCIES: 'oldest' }
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@seberg @vyasr @jameslamb Do we need CI coverage for "oldest dependencies + latest Python version"? I think it might be good to cover that somewhere in our CI to ensure that our pinnings are compatible with all the Python versions we support, but I'm not sure whether others think that is necessary.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would lean towards not worrying too much about it and not do it if that is the only purpose of the job. I doubt it is a very common setup, and while I think having some tests with the oldest versions is important, I doubt that covering it for all Python versions is.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could go either way on this... right now I'm weakly in favor of adding an "oldest dependencies + latest Python version" job right for nightlies.

To help myself think through this, I wrote out some specific examples below.

Problems I can envision such a job catching that none of the other CI jobs catch (for a fictional dependency somedep):

  1. somedep==1.2.0 does not have wheels for python={newest}
    • which we only catch in CI because compilation of an sdist fails, or there isn't an sdist available at all
  2. somedep==1.2.0 uses some parts of the Python standard library that were changed in some breaking way by python={newest}
    • most likely when somedep publishes py3-none wheels (pure Python like dask or not linking against Python like xgboost)
  3. somedep==1.2.0 has a restrictive pin on Python-requires

A CI job that exposes these types of things could inform more detailed pins for wheels, like this:

dependencies = [
  "somedep>=1.2.0 ; python_version < '3.10'",
  "somedep>=1.4.1 ; python_version >= '3.10'"
]

That richer detail is useful for folks using uv's support for solving to the oldest compatible versions (#228 (comment)) or for anyone in an environment where library versions are locked (e.g. with pip freeze) and who want to upgrade as rarely and as slowly as possible.

I've done exactly that before, actually... I worked on a team managing a medium-sized Apache Airflow deployment where a couple dozen Python codebases had to all share a single Python version... and share it with the service itself. When we went to upgrade the Python version in that system, we tried to upgrade as few libraries as possible, and to update them no further than absolutely necessary to support the next version of Python. In that exercise, I was happy to find out about issues like these via failed pip solves... and sad to find out about them at runtime 😅

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Talked about this with @bdice offline.

I still support adding something like it.... but not for this release. We're getting close to burndown for the 24.10 release and trying to get a large amount of packaging changes in.

I think the matrix proposed here should be left as-is and we could come back to testing "latest Python + oldest dependencies" at some time in the future.

- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '12.0.1', LINUX_VER: 'rockylinux8', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
# arm64
- { ARCH: 'arm64', PY_VER: '3.10', CUDA_VER: '12.2.2', LINUX_VER: 'ubuntu22.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'oldest' }
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu22.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '12.5.1', LINUX_VER: 'rockylinux8', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.12', CUDA_VER: '12.5.1', LINUX_VER: 'rockylinux8', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
"

TEST_MATRIX=$(yq -n 'env(MATRICES) | .[strenv(BUILD_TYPE)]')
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/wheels-build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -86,11 +86,15 @@ jobs:
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '12.5.1', LINUX_VER: 'rockylinux8' }
- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '11.8.0', LINUX_VER: 'rockylinux8' }
- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '12.5.1', LINUX_VER: 'rockylinux8' }
- { ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '11.8.0', LINUX_VER: 'rockylinux8' }
- { ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '12.5.1', LINUX_VER: 'rockylinux8' }
# arm64
- { ARCH: 'arm64', PY_VER: '3.10', CUDA_VER: '11.8.0', LINUX_VER: 'rockylinux8' }
- { ARCH: 'arm64', PY_VER: '3.10', CUDA_VER: '12.5.1', LINUX_VER: 'rockylinux8' }
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '11.8.0', LINUX_VER: 'rockylinux8' }
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '12.5.1', LINUX_VER: 'rockylinux8' }
- { ARCH: 'arm64', PY_VER: '3.12', CUDA_VER: '11.8.0', LINUX_VER: 'rockylinux8' }
- { ARCH: 'arm64', PY_VER: '3.12', CUDA_VER: '12.5.1', LINUX_VER: 'rockylinux8' }
"

MATRIX="$(
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/wheels-test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -82,16 +82,16 @@ jobs:
# amd64
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'oldest' }
# arm64
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu20.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.12', CUDA_VER: '11.8.0', LINUX_VER: 'ubuntu20.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
nightly:
# amd64
- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '11.8.0', LINUX_VER: 'rockylinux8', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'oldest' }
- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '12.0.1', LINUX_VER: 'ubuntu20.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
# arm64
- { ARCH: 'arm64', PY_VER: '3.10', CUDA_VER: '11.8.0', LINUX_VER: 'rockylinux8', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'oldest' }
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '12.2.2', LINUX_VER: 'ubuntu20.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.11', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
- { ARCH: 'arm64', PY_VER: '3.12', CUDA_VER: '12.5.1', LINUX_VER: 'ubuntu22.04', GPU: 'a100', DRIVER: 'latest', DEPENDENCIES: 'latest' }
"

TEST_MATRIX=$(yq -n 'env(MATRICES) | .[strenv(BUILD_TYPE)]')
Expand Down