[CI] Add K8s Builder Step

[simon-mo]

Why are these changes needed?

This PR bring in a kind cluster with our current CI infra and verify it works.

Related issue number

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[DmitriGekhtman]

Hey, this is great!

Some stupid questions:

• Any particular reasons for choice of kind vs minikube for this
• Does this build step itself happen in a containerized environment (and if so is there any trouble running kind in a container)
• Is it possible to get an indication of the CPU and memory capacity of the single Kubernetes node set up here (and pointers for how to configure that in the future)?

[simon-mo]

The build step run inside a container. So we can't use minikube directly, minikube relies on virtual machine. We have already setup a docker-in-docker environment inside a separate container (hence the mentioning of dind). This PR runs kind inside that environment and allow the host container (where build steps run) talk to the the K8s api server.

It's a bit crazy but basically we have the following architecture

The kind cluster should be able to use containers from that dind nodes if we configure local registry https://kind.sigs.k8s.io/docs/user/local-registry/, I'll probably do it in a separate PR.

There isn't any limit for the CPU or memory capacity, but in the end the whole thing is running on a single VM. We can run it on a bigger box though.

[simon-mo]

Actually we can use kind load docker-images: https://kind.sigs.k8s.io/docs/user/quick-start/#loading-an-image-into-your-cluster

[DmitriGekhtman]

Cool, got it (mostly). Which container does the test code run in?
Say, for example, I have a test that wants to start a Ray-on-K8s cluster and connect to it "from outside" via port-forwarding and Ray client. How does it work?

[simon-mo]

Which container does the test code run in?

The left hand side host container

Say, for example, I have a test that wants to start a Ray-on-K8s cluster and connect to it "from outside" via port-forwarding and Ray client. How does it work?

Node port should work as long as you use docker as DNS name.