Add capability to exclude directories/file types from file_mounts within cluster YAML

[richardrl]

Why are these changes needed?

Small syntax error fixes that prevent running various functions like "ray submit" successfully and add functionality for excluding files when syncing directories to clusters.

Checks

• I've run scripts/format.sh to lint the changes in this PR. -> No, I get this error and I don't know what it means:
  flake8: error: unrecognized arguments: --inline-quotes --no-avoid-escape python/ray/autoscaler/autoscaler.py python/ray/autoscaler/command_runner.py python/ray/autoscaler/commands.py python/ray/autoscaler/node_provider.py python/ray/autoscaler/updater.py python/ray/scripts/scripts.py python/ray/tune/suggest/suggestion.py python/ray/tune/syncer.py
• I've included any doc changes needed for https://docs.ray.io/en/latest/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failure rates at https://ray-travis-tracker.herokuapp.com/.
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested (please justify below)
    Not sure what to run. Are there explicit instructions anywhere?

[richardrl]

To use, add like this to YAML (rsync exclude syntax):

exclude_list:
 - "out/"
 - ".git/"
 - "histograms/"

[ijrsvt]

@richardrl To run ./scripts/format.sh first install the following packages

flake8==3.7.7 
flake8-comprehensions
flake8-quotes==2.0.0
yapf==0.23.0

[ijrsvt]

Is this just for debugging?

[ijrsvt]

Great idea for a PR--this will be super useful to reduce the number of extraneous files being copied (especially caches and .git files). The only thing is it would be great to try to keep this option just for the relevant commands (rsync). If you have any questions--please let me know in follow-ups :)

[ijrsvt]

Could you possibly include this as an argument to run_rsync_up and run_rsync_down instead of the constructors?

[ijrsvt]

Specifically, can you pas in the exclude_list into to the rsync function and the updater.sync_file_mounts function.

Since rsync is just updater.rsync_down or updater.rsync_up, those two functions need to be modified.

[ijrsvt]

Oh, can you also pull the latest master locally and then merge master to hide all the changes already committed?

[bveeramani]

‼️ ACTION REQUIRED ‼️

We've switched our code formatter from YAPF to Black (see #21311).

To prevent issues with merging your code, here's what you'll need to do:

1. Install Black

pip install -I black==21.12b0

2. Format changed files with Black

curl -o format-changed.sh https://gist.githubusercontent.com/bveeramani/42ef0e9e387b755a8a735b084af976f2/raw/7631276790765d555c423b8db2b679fd957b984a/format-changed.sh
chmod +x ./format-changed.sh
./format-changed.sh
rm format-changed.sh

3. Commit your changes.

git add --all
git commit -m "Format Python code with Black"

4. Merge master into your branch.

git pull upstream master

5. Resolve merge conflicts (if necessary).

After running these steps, you'll have the updated format.sh.

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

• If you'd like to keep this open, just leave any comment, and the stale label will be removed.

[stale]

Hi again! The issue will be closed because there has been no more activity in the 14 days since the last message.

Please feel free to reopen or open a new issue if you'd still like it to be addressed.

Again, you can always ask for help on our discussion forum or Ray's public slack channel.

Thanks again for opening the issue!