Install Dask + Distributed from main

[jakirkham]

These recently dropped the master branch and switched to main. So update the install steps to use main instead.

[jakirkham]

@gpucibot merge

[jakirkham]

Seems like we are getting some explicit comms test failures. @rjzamora would you be able to take a look? 🙂

[pentschev]

The issues seem to be GPU-related in CI:

22:30:30 [100] Call to cuInit results in CUDA_ERROR_NO_DEVICE:
22:30:31 Coverage.py warning: --include is ignored because --source is set (include-ignored)
22:30:31 Coverage.py warning: --include is ignored because --source is set (include-ignored)
22:30:31 Unable to start CUDA Context
22:30:31 Traceback (most recent call last):
22:30:31   File "/opt/conda/envs/rapids/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 237, in initialize
22:30:31     self.cuInit(0)
22:30:31   File "/opt/conda/envs/rapids/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 300, in safe_cuda_api_call
22:30:31     self._check_error(fname, retcode)
22:30:31   File "/opt/conda/envs/rapids/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 335, in _check_error
22:30:31     raise CudaAPIError(retcode, msg)
22:30:31 numba.cuda.cudadrv.driver.CudaAPIError: [100] Call to cuInit results in CUDA_ERROR_NO_DEVICE

Rerunning to see if this unblocks.

[pentschev]

rerun tests

[jakirkham]

Seeing this in the log

10:58:40   File "/opt/conda/envs/rapids/lib/python3.7/site-packages/ucp/core.py", line 628, in recv
10:58:40     ret = await comm.tag_recv(self._ep, buffer, nbytes, tag, name=log)
10:58:40 ucp.exceptions.UCXMsgTruncated: <[Recv #112] ep: 0x7f0e25cde0d8, tag: 0xfa85496d273cdec2, nbytes: 260, type: <class 'numpy.ndarray'>>: length mismatch: 16 (got) != 260 (expected)

Also seeing this

11:05:40   File "/var/lib/jenkins/workspace/rapidsai/gpuci/dask-cuda/prb/dask-cuda-gpu-test/CUDA/10.1/GPU_LABEL/gpu-t4||gpu/OS/ubuntu16.04/PYTHON/3.7/dask_cuda/explicit_comms/dataframe/shuffle.py", line 196, in local_shuffle
11:05:40     out_parts[i] = None
11:05:40 TypeError: 'tuple' object does not support item assignment
11:05:40 FAILED

[jakirkham]

Wondering if that last part is related to PR ( dask/distributed#4531 )

cc @madsbk (in case you have any thoughts here 🙂)

[pentschev]

Ah sorry @jakirkham , I missed those errors, but I'm seeing them now too on the latest CI run as well. Seems like a potential issue coming from dask/distributed#4531 indeed, so it would be good to have @madsbk looking into it.

To unblock CI, what do you think about xfailing those tests @jakirkham ?

[jakirkham]

Sure marked as xfail. Raised issue ( #549 ) to track and included this in the xfail message

[pentschev]

LGTM, thanks @jakirkham !

[pentschev]

@jakirkham seems like

dask-cuda/dask_cuda/tests/test_explicit_comms.py

Line 248 in f520186

def test_dask_use_explicit_comms():

should also be xfailed. Could you do that as well?

[jakirkham]

Good catch! Thanks Peter 😄

Sorry had missed that earlier. Should be addressed now 🙂