[ci] Move to new hierarchical docker structure + pipeline

[krfricke]

Why are these changes needed?

This PR moves our buildkite pipeline to a new hierarchical structure and will be used with the new buildkite pipeline.

When merging this PR, the old behavior will still work, i.e. the old pipeline is still in place.

After merging this PR, we can build the base images for the master branch, and then switch the CI pipelines to use the new build structure.

Once this switch has been done, the following files will be removed:

• ./buildkite/pipeline.yml - this has been split into pipeline.test.yml and pipeline.build.yml
• ./buildkite/Dockerfile - this has been moved (and split) to ./ci/docker/
• ./buildkite/Dockerfile.gpu - this has been moved (and split) to ./ci/docker/

The new structure is as follow:

• ./ci/docker contains hierarchical docker files that will be built by the pipeline.
• Dockerfile.base_test contains common dependencies
• Dockerfile.base_build inherits from it and adds build-specific dependencies, e.g. llvm, nvm, java
• Dockerfile.base_ml inherits from base_test and adds ML dependencies, e.g. torch, tensorflow
• Dockerfile.base_gpu depends on a cuda image and otherwise has the same contents as base_test and base_ml combined

In each build, we do the following

• Dockerfile.build is built on top of Dockerfile.base_build. Dependencies are re-installed, which is mostly a no-op (except if they changed from when the base image was built)
• Dockerfile.test is built on top of Dockerfile.base_test, and the extracted Ray installation fromDockerfile.build is injected
• The same is true respectively for ml and gpu.

The pipelines have been split, and a new attribute NO_WHEELS_REQUIRED is added, identifying tests that can be early-started. Early start means that the last available branch image is used and the current code revision is checked out upon it.

See https://github.com/ray-project/buildkite-ci-pipelines/ for the pipeline logic.

Additionally, this PR identified two CI regressions that haven't been caught previously, namely the minimal install tests that didn't properly install the respective Python versions, and some runtime environment tests that don't work with later Ray versions. These should be addressed separately and I'll create issues for them once this PR is merged.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[simon-mo]

Looks great overall, three small nits:

• We should still run the tests in 3.6 right? Did you end up making a special build for 3.6? If we still support 3.6, we should run all libraries test in it. (instead of bumping the base image python version)
• Please add some readmes in several places (.buildkite, ci/, and the pipelines repo: https://github.com/ray-project/buildkite-ci-pipelines/tree/main/ray_ci)
• Dockerfile.xyz naming seems to confuse GitHub and code editor, can we use xzy.Dockerfile for the new files.

[krfricke]

Thanks for the review!

• I've renamed the dockerfiles
• I've added READMEs in ci/docker and .buildkite

For the 3.6 tests, we've been running over 75% of the tests on 3.7 already. My suggestion here is to go ahead with this change and use #28373 to introduce 2 or 3 build jobs that ensure 3.6 compatibility for specific tests.

I'll update #28373 after this has been merged.