[data] Optimize dataset metadata read/write in Ray client

[ericl]

Why are these changes needed?

In Ray client mode, the client may not have the same low-latency access to the datasource as the cluster nodes. Run datasource prepare_read/do_write in a remote task to make sure it doesn't run on the driver node.

Related issue number

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[jjyao]

Hmm.. does that mean we cannot really do anything heavy in the driver if ray client is a first class citizen.

[jjyao]

What's the cost for non-ray client mode?

[ericl]

Shouldn't matter much (a few ms)

[ericl]

Yeah I think that's a good bias for Ray native libraries (for example, Tune already avoids running heavy things in the driver).

[jjyao]

Yeah I think that's a good bias for Ray native libraries (for example, Tune already avoids running heavy things in the driver).

I didn't have that awareness before but yeah I (really everyone) will keep that in mind for future code. Not sure if there is something helps us to remember this (probably one item on the review action list is checking to see if the code works nicely with ray client).

[ericl]

Hmm this seems to be causing test_basic_actors to hang somehow.

[ericl]

Blocked on #21970

[bveeramani]

‼️ ACTION REQUIRED ‼️

We've switched our code formatter from YAPF to Black (see #21311).

To prevent issues with merging your code, here's what you'll need to do:

1. Install Black

pip install -I black==21.12b0

2. Format changed files with Black

curl -o format-changed.sh https://gist.githubusercontent.com/bveeramani/42ef0e9e387b755a8a735b084af976f2/raw/7631276790765d555c423b8db2b679fd957b984a/format-changed.sh
chmod +x ./format-changed.sh
./format-changed.sh
rm format-changed.sh

3. Commit your changes.

git add --all
git commit -m "Format Python code with Black"

4. Merge master into your branch.

git pull upstream master

5. Resolve merge conflicts (if necessary).

After running these steps, you'll have the updated format.sh.