[ci/gpu] Move ML dependency install to Dockerfile

[krfricke]

Why are these changes needed?

Instead of installing dependencies in each Buildkite job, let's move this to the Dockerfile instead.
This will update GPU tests to always use Python 3.7.

Cc @richardliaw @amogkam @matthewdeng will updating to 3.7 be a problem? Most GPUs tests are already installing Python 3.7, the exception are the Train tests.

Related issue number

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[matthewdeng]

3.7 should be fine for Train!

[matthewdeng]

I'm of the opinion that we should try to keep dependency installation to be the minimal set needed for each test suite.

If we always install the superset of dependencies, our tests may not be reflective of real user setups and may miss some cases. Though perhaps for GPU we do intentionally want to include at least the ML docker image dependencies.... thoughts?

[krfricke]

The goal of moving the installs into the Dockerfile is to speedup subsequent pipeline steps, e.g. reduce the cumulative installation time. The dependencies that benefit the most from this are presumably large dependencies, like tensorflow and torch.

The decision we'll have to make here is if we care about testing a minimal setup, and if we want to include these checks in the CI pipeline. We do have a minimal dependency test step, but it only tests a small subset at the moment.

That said, I agree that we should probably limit the default dependencies here.

Actually, let's hold off this PR for now, I think I have an idea on how to generally solve this problem.

[matthewdeng]

Should this be implemented directly within install-dependencies.sh? Seems weird to have the option to specify a PYTHON version but also needing to do this.

[krfricke]

Many tests currently separately call ./ci/travis/ci.sh build. You are right that this is not ideal, but I would say let's keep it out of the scope of this PR and review this in the future when refactoring CI structure

[matthewdeng]

Since the default version is now 3.7, should PYTHON=3.7 be removed?

[krfricke]

I'd keep it here for verbosity

[bveeramani]

‼️ ACTION REQUIRED ‼️

We've switched our code formatter from YAPF to Black (see #21311).

To prevent issues with merging your code, here's what you'll need to do:

1. Install Black

pip install -I black==21.12b0

2. Format changed files with Black

curl -o format-changed.sh https://gist.githubusercontent.com/bveeramani/42ef0e9e387b755a8a735b084af976f2/raw/7631276790765d555c423b8db2b679fd957b984a/format-changed.sh
chmod +x ./format-changed.sh
./format-changed.sh
rm format-changed.sh

3. Commit your changes.

git add --all
git commit -m "Format Python code with Black"

4. Merge master into your branch.

git pull upstream master

5. Resolve merge conflicts (if necessary).

After running these steps, you'll have the updated format.sh.