[FEA] Enable test parallelism

[csadorf]

As reported by @vyasr there are currently some issues with running tests in paralle using pytest-xdist. We should resolve those to significantly reduce cuml's test runtime when multiple workers are available.

• Ensure consistent order for nearest neighbor tests #5059

[csadorf]

@vyasr Maybe you can provide a little bit more context?

[vyasr]

Yup, the current issue that I run into is that the different workers collect different tests. Offhand I would guess that the problem is one of the following:

1. Some unordered container is being used to parametrize tests and is resulting in different orders on different workers.
2. Some sort of randomization is being used in parametrization to select which tests run when
3. There is something nondeterministic in the selection of stress/quality/unit tests that changes their ordering.

I've attached the output that I see.

pytest_output_two_workers.txt

[wphicks]

The non-determinism in test collection should be fixed by #5059, but it looks like we've got issues with some of the tests themselves unrelated to the question of parallelism.

[vyasr]

Could you clarify what issues you're talking about? Are these issues that prevent the usage of parallelism (perhaps because of GPU overutilization or memory requirements), unrelated issues that lead to long test times (aka #5053), or something else?

[wphicks]

Sorry, I was actually referring to unrelated failures that were present across PRs. That has been resolved, but now there are indeed failures which look like they're definitely caused by the parallelism itself. Specifically, we're seeing:

• Hypothesis tests exceeding their deadlines (probably benign resource contention and easily resolvable)
• Running out of GPU memory
• Some more troubling failures in the Dask tests, including what appears to be actual numerical errors

[dantegd]

xdist is now being used in the GHA based CI