[Feature] Add Helm Chart Checker in CI

[Jeffwan]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

This is requested from @chenk008 in #175 (comment).

We maintain both the kustomize and helm manifests. Currently, we do use kustomize to deploy the control plane and validate cluster creations etc in CI. However, we lack of tests to validate helm chart changes. This can be improved by adding helm chart checker in CI.

/cc @ebr @Chenk

Use case

Validate helm manifest changes make sure it's working

Related issues

#175

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[DmitriGekhtman]

It's partially covered by #554.

What remains to be done is to add KinD CI integration tests that validate that the charts work end-to-end.

[DmitriGekhtman]

What remains to be done is to add KinD CI integration tests that validate that the charts work end-to-end.

A basic e2e sanity check that we can use Helm to deploy a 1-head and 1-worker Ray cluster with the latest Ray version would be sufficient.

[DmitriGekhtman]

We can use existing compatibility test code as a starting point for this.

[DmitriGekhtman]

@kevin85421 for the KubeRay 0.4.0 release we should add a e2e CI test that validates that at least the default values.yaml works

[DmitriGekhtman]

If we can't add it to CI before 0.4.0, we should at least test manually.

[DmitriGekhtman]

Again, not a hard blocker for the 0.4.0 release, but we do need to at least do some manual testing.