[Dask-on-Ray] Add Dask config helper, set task-based shuffle by default.

[clarkzinzow]

Dask default's to a disk-based shuffle even thought we're using a distributed scheduler, which appears to be resulting in dropped data since the filesystem isn't shared across nodes. Dask Distributed manually sets the shuffle algorithm in the global config to the task-based shuffle, which the Dask-on-Ray scheduler should probably do as well.

This PR adds a Dask config helper, enable_dask_on_ray, that sets Dask-on-Ray as the default scheduler along with changing the default shuffle to a task-based shuffle. The shuffle method can still be overridden by the user by manually specifying df.set_index(shuffle="disk").

Related issue number

Closes #20108, closes #20992, closed #14048

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[ericl]

Looks good. I've always wanted this override to happen by default!

[clarkzinzow]

@ericl Once I saw that Dask Distributed was doing it, I felt a lot better about it. 😄 Although they still have the option Client(set_as_default=False) to keep from setting the global config, but I think that we can wait to expose that until a user asks for it.

[rkooo567]

Nice job finding this. Is there a way to add a test?

[ericl]

//python/ray/util/dask:test_dask_optimization FAILED in 3 out of 3 in 9.9s

[rkooo567]

Request change to add test cases?

[clarkzinzow]

@rkooo567 Sure, I guess I could remove the explicit setting of the scheduler in the scheduler tests now that it happens automatically at import time.

[rkooo567]

Does this mean we allow to disable this in some cases? If so, I think we should have a test for that (disabling case)

[clarkzinzow]

@rkooo567 @ericl changed it to a config-setting helper, enable_dask_on_ray, which we can encourage Dask-on-Ray users to use.

[rkooo567]

Lgtm!!

[ericl]

I'm also OK with this approach.