add support for python 3.11

[jameslamb]

Contributes to rapidsai/build-planning#3.

Proposes adding Python 3.11 entries in the following workflows:

• conda-python-build
• conda-python-test
• wheels-build
• wheels-test

Notes for Reviewers

Why these specific matrix entries?

Following the constraints laid out in the description of rapidsai/build-planning#3, and the discussion from #166, I'm proposing:

• adding Python 3.11 builds for every combination where we currently do 3.10 builds (since those don't require GPUs)
• replacing some Python 3.10 tests with Python 3.11
  • holding the absolute number of testing jobs unchanged
  • trying to spread Python 3.11 test jobs across CUDA version, operating system, and architecture
  • trying to cover "oldest of everything"
  • trying to cover "latest of everything"

Rollout plan?

• 1. merge this PR

I'm proposing using GitHub's continue-on-error (docs) to allow the Python 3.11 jobs to fail on PRs without blocking those PRs from being merged.

So this could be merged into branch-24.04 without making updates in all the repos.

• 2. update all the repos, in RAPIDS dependency order

Using something like https://github.com/rapidsai/rapids-reviser/pull/11, go update all the repos... adding Python 3.11 blocks to dependencies.yaml, updating pyproject.toml and docs, and getting these CI jobs all passing.

• 3. make these jobs required

Once all RAPIDS repos have been updated and these. Python 3.11 jobs are passing for them, making these jobs required to prevent further development from breaking Python 3.11 support

• 4. other steps in "Post Migration" described at Add support for Python 3.11 build-planning#3
• 5. switch to Python 3.11 in wheels-publish CI job
  • run wheels-publish jobs in Python 3.11 image #169

How I tested this

Put up rapidsai/rmm#1469, with extra changes pointing its GitHub Actions configuration at this branch. Saw all the Python 3.11 jobs expected get run (and they happened to run successfully, yay!). Looked through the logs and confirmed that Python 3.11 was really being pulled in there, and that wheels had names like rm-*-cp311-*.whl.

Also put up rapidsai/cudf#15111, to test a more complex pipeline. Saw all the Python 3.11 jobs expected get run... many of them failed, but for the expected types of reasons... for example, there weren't rmm-cu12==24.4.* wheels supporting Python 3.11 available yet (build link).

But to be clear, I only changed those GitHub Actions configs to point to this branch to test this PR. That shouldn't be necessary for any of the other PRs, if we go with the continue-on-error approach I'm proposing here.

[KyleFromNVIDIA]

Looks good overall, would like to see TODO comments explaining the continue-on-error directives.

[msarahan]

adding Python 3.11 builds for every combination where we currently do 3.11 builds (since those don't require GPUs)
I'm pretty sure you mean add 3.11 everywhere that we currently have 3.10, but just want to make certain.

I don't like the approach of keeping the same number of tests, but just assigning different combinations. I would like it better if we had random sampling without replacement. I googled around for that quickly, and it seems possible but not worth the effort. The approach here is probably good enough, even if it feels a little off.

[bdice]

Let’s reshuffle these a bit. I want one configuration to be amd64 + latest Python + latest CUDA. Currently it seems only ARM has Python 3.11 jobs. I would also tilt the distribution of versions to be evenly weighted, it seems like the matrix tests 3.9 and 3.10 more than 3.11 (I didn’t do an exact count).

[jameslamb]

Thanks for looking @bdice .

This PR does currently have a Python 3.11 AMD job for wheels-test... but just on PRs.

- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '12.2.2', LINUX_VER: 'ubuntu20.04', gpu: 'v100', driver: 'latest' }

Definitely agree the mix across Python versions is uneven right now, I'll fix that.

Before I go further with this... would you consider this change I just put up?

#177

Based on my experience here and in #166, I think it's worth taking a minute to re-sorting these matrices so they're easier to review.

[jameslamb]

Alright @bdice whenever you have time, could you please take another look?

(and @msarahan @AyodeAwe @KyleFromNVIDIA if you'd like another chance to review now that changes have been made)

By my count, on the current state of this branch we have the following mix:

workflow	python version	amd64	arm64	total
conda-python-build	3.9	2	2	4
conda-python-build	3.10	2	2	4
conda-python-build	3.11	2	2	4
conda-python-tests (pr)	3.9	1	0	1
conda-python-tests (pr)	3.10	1	1	2
conda-python-tests (pr)	3.11	1	0	1
conda-python-tests (nightly)	3.9	2	1	3
conda-python-tests (nightly)	3.10	2	1	3
conda-python-tests (nightly)	3.11	1	2	3
wheel-build	3.9	4	2	6
wheel-build	3.10	4	2	6
wheel-build	3.11	4	2	6
wheel-test (pr)	3.9	1	0	1
wheel-test (pr)	3.10	0	1	1
wheel-test (pr)	3.11	1	0	1
wheel-test (nightly)	3.9	2	1	3
wheel-test (nightly)	3.10	2	2	4
wheel-test (nightly)	3.11	2	1	3

And the conda-python-tests (pr) and wheel-test (pr) both have the combination "amd64 + latest Python + latest CUDA".

I've still preserved the same absolute number of test jobs as branch-24.04.

I've also updated the description, with tests from these two PRs:

• Add support for Python 3.11 rmm#1469
• Add support for Python 3.11, require NumPy 1.23+ cudf#15111

[bdice]

It looks like we're not testing amd64 + latest CUDA + latest Python. I want that specific configuration to be in the matrix since it is a helpful edge case. This is what I would have expected (swap CUDA versions):

Suggested change

	- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '12.2.2', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest' }
	- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '12.0.1', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest' }
	- { ARCH: 'amd64', PY_VER: '3.10', CUDA_VER: '12.0.1', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest' }
	- { ARCH: 'amd64', PY_VER: '3.11', CUDA_VER: '12.2.2', LINUX_VER: 'ubuntu22.04', GPU: 'v100', DRIVER: 'latest' }

[jameslamb]

Promise I didn't ignore your previous comment asking for that combination.

On the current state of this branch, that exact combination is being tested on PRs for conda-python-tests workflow, which is why I didn't also add it to the nightly matrix.

Does that change you opinion on this suggestion?

Or do you think it's desirable to have that combination in both places because we don't enforce that PRs have to be up to date with the target branch before merging?

[bdice]

Or do you think it's desirable to have that combination in both places because we don't enforce that PRs have to be up to date with the target branch before merging?

(Yes, and...) it's very helpful for the nightly matrix to be a strict superset of the PR matrix. It's hard to test things that fail in nightlies unless we have a corresponding entry in the PR matrix (otherwise we're stuck merging and waiting for another nightly). This comes at a slight cost to overall matrix-coverage-over-time but it really helps the sanity of developers. 😅

And even if the PR matrix is not a strict subset of the nightly matrix -- it helps for it to be pretty close. I definitely want the "endpoints" like latest-everything to be in both matrices.

[jameslamb]

Ok got it! That was not a constraint I was considering when I came up with the proposal here. I was leaning toward denser coverage of the matrix over time in exchange for a higher risk of "this PR passed but nightly builds failed", since I saw that the nightly matrix was already so much bigger than the PR one.

I've added a comment about that at rapidsai/build-planning#5 (comment) ... think it's another example of the type of constraint we should be able to enforce with automation here, to reduce the effort it requires for you and others to review PRs like this one.

I'll go over all the matrices touched in this PR and make sure this constraint's being met, will request another review when that's done.

[jameslamb]

Just pushed 115ad5a. As of that commit, I believe this branch is meeting these constraints for test job matrices:

• at least 1 PR job for every Python version in (3.9, 3.10, 3.11)
• at least 1 PR job for every CUDA version in (11.8.0, 12.0.1, 12.2.2)
• at least 1 PR job for every architecture in (amd64, arm64)
• am64 + latest CUDA + latest Python + latest Ubuntu present in both PR and nightly
• nightly is a strict superset of what's tested on PRs
• no net change to the number of test jobs on PRs or nightly, relative to the current state of branch-24.04
• as-close-as-possible-to-equal spread of test jobs across Python versions

Since that commit only changed CUDA_VER and LINUX_VER, the table from #176 (comment) is unchanged.

[bdice]

Note that when we drop CentOS 7 (glibc 2.17) in RAPIDS 24.06, we will also have to drop all our test configurations with Ubuntu 18.04 (glibc 2.27). Our new minimum glibc version will be 2.28.

We have already dropped Ubuntu 18.04 support but we will need to follow-up and remove it from our CI matrices once our minimum glibc is bumped.

All this is now documented here: rapidsai/build-planning#23

[vyasr]

adding Python 3.11 builds for every combination where we currently do 3.11 builds (since those don't require GPUs)
I'm pretty sure you mean add 3.11 everywhere that we currently have 3.10, but just want to make certain.

I don't like the approach of keeping the same number of tests, but just assigning different combinations. I would like it better if we had random sampling without replacement. I googled around for that quickly, and it seems possible but not worth the effort. The approach here is probably good enough, even if it feels a little off.

I'd say we stick with the approach in this PR for now. Once this PR merges, I plan to seriously evaluate changes to address rapidsai/build-planning#5

[vyasr]

I'm comfortable with the choices currently made in this PR for now. Once this merges, I'd like to focus on addressing rapidsai/build-planning#5 in a PR where we can relax constraints around the number of test jobs (I agree with conserving that number in this PR to avoid doing too many things at once).