test_cuda_visible_devices_and_memory_limit_and_nthreads spews benign (?) warnings on systems with fewer than eight GPUs

[wence-]

          > > it doesn't look like it failed any tests though. Is this a problem?

This looks bad to me. I wonder if it is happening not just on this branch, I will investigate

On closer examination, this error in the test logs comes from this test

dask-cuda/dask_cuda/tests/test_dask_cuda_worker.py

Lines 25 to 61 in 8134e6b

	@patch.dict(os.environ, {"CUDA_VISIBLE_DEVICES": "0,3,7,8"})
	def test_cuda_visible_devices_and_memory_limit_and_nthreads(loop): # noqa: F811
	nthreads = 4
	with popen(["dask", "scheduler", "--port", "9359", "--no-dashboard"]):
	with popen(
	[
	"dask",
	"cuda",
	"worker",
	"127.0.0.1:9359",
	"--host",
	"127.0.0.1",
	"--device-memory-limit",
	"1 MB",
	"--nthreads",
	str(nthreads),
	"--no-dashboard",
	"--worker-class",
	"dask_cuda.utils.MockWorker",
	]
	):
	with Client("127.0.0.1:9359", loop=loop) as client:
	assert wait_workers(client, n_gpus=4)
	def get_visible_devices():
	return os.environ["CUDA_VISIBLE_DEVICES"]
	# verify 4 workers with the 4 expected CUDA_VISIBLE_DEVICES
	result = client.run(get_visible_devices)
	expected = {"0,3,7,8": 1, "3,7,8,0": 1, "7,8,0,3": 1, "8,0,3,7": 1}
	for v in result.values():
	del expected[v]
	workers = client.scheduler_info()["workers"]
	for w in workers.values():
	assert w["memory_limit"] == MEMORY_LIMIT // len(workers)

Which is written assuming eight GPUs are available on the system running the test. So I think these problems in the logs are benign, but I will open a separate PR to fix this latter problem for the 23.04 branch.

Originally posted by @wence- in #1123 (comment)

[pentschev]

This is intentional, we're testing the ability to set CUDA_VISIBLE_DEVICES appropriately for each worker, and as noticed the downside are those errors that are harmless when fewer GPUs are available, but the logic test is still successful nevertheless. We could eventually attempt to ingest the warnings and suppress them instead, but other than that we don't want to remove the test or dumb it down only to prevent the stdout/stderr output.

[wence-]

My proposal would be to try and capture the warnings if the number of devices is fewer than required (rather than dumbing down the test).

[pentschev]

Yes, that would be ideal in this situation.