Support CUDA 12.2

[jameslamb]

Description

• switches to CUDA 12.2.2 for building conda packages and wheels
• adds new tests running against CUDA 12.2.2
• adds some "12.*" globs in dependencies.yaml missed in refactor CUDA versions in dependencies.yaml #1317

Notes for Reviewers

This is part of ongoing work to build and test packages against CUDA 12.2.2 across all of RAPIDS.

For more details see:

• Add support for CUDA 12.2 wheels build-planning#7
• add CUDA 12.2 conda and wheel jobs shared-workflows#166

Planning a second round of PRs to revert these references back to a proper branch-24.{nn} release branch of shared-workflows once rapidsai/shared-workflows#166 is merged.

(created with rapids-reviser)

[jameslamb]

After merging in latest branch-24.04, one build (conda-python-tests (12.0.1, ubuntu22.04, arm64, 3.10, a100)) is failing like this:

self = <numba.cuda.cudadrv.driver.CtypesLinker object at 0xfffe56f4c790>
ptx = b'//\n// Generated by NVIDIA NVVM Compiler\n//\n// Compiler Build ID: CL-33567101\n// Cuda compilation tools, release ...t%rd84, %rd83, %rd82;\n\tld.global.u64 \t%rd85, [%rd74];\n\tst.global.u8 \t[%rd84], %rd85;\n\n$L__BB0_27:\n\tret;\n\n}'
name = '<cudapy-ptx>'

    def add_ptx(self, ptx, name='<cudapy-ptx>'):
        ptxbuf = c_char_p(ptx)
        namebuf = c_char_p(name.encode('utf8'))
        self._keep_alive += [ptxbuf, namebuf]
        try:
            driver.cuLinkAddData(self.handle, enums.CU_JIT_INPUT_PTX,
                                 ptxbuf, len(ptx), namebuf, 0, None, None)
        except CudaAPIError as e:
>           raise LinkerError("%s\n%s" % (e, self.error_log))
E           numba.cuda.cudadrv.driver.LinkerError: [222] Call to cuLinkAddData results in UNKNOWN_CUDA_ERROR
E           ptxas application ptx input, line 9; fatal   : Unsupported .version 8.3; current version is '8.2'

/opt/conda/envs/test/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py:2830: LinkerError
------------------------------ Captured log call -------------------------------
ERROR    numba.cuda.cudadrv.driver:driver.py:396 Call to cuLinkAddData results in UNKNOWN_CUDA_ERROR

(build link)

I don't see a similar error in other recent runs of the pr workflow here: https://github.com/rapidsai/cuspatial/actions/workflows/pr.yaml

I found numba/numba#8961 (comment) which suggests that maybe this error is about a mismatch between the driver version and the PTX numba is using. That corresponds with this warning I see in the build:

/opt/conda/envs/test/lib/python3.10/site-packages/cudf/utils/_numba.py:18: UserWarning: CUDA Toolkit is newer than CUDA driver. Numba features will not work in this configuration.

nvidia-smi output from that failing job shows that v535 of the driver is being used.

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100-PCIE-40GB          On  | 00000000:01:00.0 Off |                    0 |
| N/A   31C    P0              32W / 250W |      4MiB / 40960MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+

I see cuda-version=12.0 being installed.

+ cuda-version                       12.0  hffde075_2                conda-forge       21kB

.... but tons of other CTK 12.3 stuff

...
+ cuda-cccl_linux-aarch64              12.3.101  h579c4fd_0                          conda-forge
+ cuda-cudart-static_linux-aarch64     12.3.101  hac28a21_0                          conda-forge
+ cuda-cudart_linux-aarch64            12.3.101  hac28a21_0                          conda-forge
+ cuda-nvvm-dev_linux-aarch64          12.3.107  h579c4fd_0                          conda-forge
+ cuda-crt-dev_linux-aarch64           12.3.107  h579c4fd_0                          conda-forge
...
+ cuda-cudart-dev_linux-aarch64        12.3.101  hac28a21_0                          conda-forge
+ cuda-nvcc-dev_linux-aarch64          12.3.107  h579c4fd_0                          conda-forge
...
+ libnvjitlink                         12.3.101  hac28a21_0                          conda-forge
...
+ cuda-nvrtc                           12.3.107  hac28a21_0                          conda-forge
+ cuda-nvvm-tools                      12.3.107  hac28a21_0                          conda-forge
+ cuda-crt-tools                       12.3.107  h579c4fd_0                          conda-forge
+ cuda-nvvm-impl                       12.3.107  hac28a21_0                          conda-forge
+ nvcomp                                  3.0.5  hed029d7_0                          conda-forge
+ cuda-cudart-static                   12.3.101  hac28a21_0                          conda-forge
+ cuda-cudart                          12.3.101  hac28a21_0                          conda-forge
...
+ libcusparse                        12.2.0.103  hac28a21_0                          conda-forge
+ libcublas                            12.3.4.1  hac28a21_0                          conda-forge
+ cuda-nvcc-tools                      12.3.107  hac28a21_0                          conda-forge
+ cuda-cudart-dev                      12.3.101  hac28a21_0                          conda-forge
+ ucx                                    1.15.0  h0461c73_3                          conda-forge
+ cuda-python                            12.2.1  py310hbbb1677_0                     conda-forge
+ numba                                  0.57.1  py310h16a1930_0                     conda-forge
...
+ libcusolver                        11.5.4.101  hac28a21_0                          conda-forge
+ cuda-nvcc-impl                       12.3.107  hac28a21_0                          conda-forge
...
+ libkvikio                           24.04.00a  cuda12_240126_g4bd5378_0            rapidsai-nightly
+ librmm                             24.04.00a5  cuda12_240126_g19d8ef96_5           rapidsai-nightly
+ rmm                                24.04.00a5  cuda12_py310_240126_g19d8ef96_5     rapidsai-nightly
+ libcudf                           24.04.00a12  cuda12_240119_ge939ee19e8_12        rapidsai-nightly
+ cudf                              24.04.00a12  cuda12_py310_240119_ge939ee19e8_12  rapidsai-nightly
+ cuproj                            24.04.00a12  cuda12_py310_240126_gab9c4de8_12    /tmp/python_channel
+ libcuspatial                      24.04.00a12  cuda12_240126_gab9c4de8_12          /tmp/cpp_channel
+ cuspatial                         24.04.00a12  cuda12_py310_240126_gab9c4de8_12    /tmp/python_channel

Interestingly, it looks like the CUDA 12.2 build is correctly getting CTK 12.2 stuff.

(build link)

My first guess is that this is about a missed condition in dependencies.yaml. The 12.0 job is the only arm64 one in the test matrix. I'll investigate that.

[jameslamb]

Ah! It looks like installing the package, cuda-version is getting upgraded!

- cuda-version                             12.0  hffde075_2                          conda-forge             Cached
+ cuda-version                             12.3  h32bc705_2                          conda-forge               21k

(build link)

I suspect this might be exactly the same issue as we observed over in cucim:

• Install cupy without explicitly requesting cuda-version now defaults to CUDA 11.8 instead of CUDA 12 conda-forge/cupy-feedstock#247
• Support CUDA 12.2 cucim#672 (comment)

[jakirkham]

Toggling for CI

[jakirkham]

Still seeing the same issue James described above. Here's the relevant snippet from CI

  - cuda-version                             12.0  hffde075_2                          conda-forge             Cached
  + cuda-version                             12.3  h32bc705_2                          conda-forge               21kB

[bdice]

Still seeing the same issue James described above. Here's the relevant snippet from CI

  - cuda-version                             12.0  hffde075_2                          conda-forge             Cached
  + cuda-version                             12.3  h32bc705_2                          conda-forge               21kB

This should be resolved by the fixes discussed in rapidsai/build-planning#8 (comment).

[jakirkham]

Indeed this is looking better. Thanks Bradley! 🙏

Next up is the Pandas 2 upgrade: #1338

[bdice]

/merge