[Feature][GCS FT] Clean up Redis once a GCS FT-Enabled RayCluster is deleted

[kevin85421]

Why are these changes needed?

• Implement Method 5 in [Bug] Deleting RayService does not clear Redis cache #1286 (comment).
• Add a feature gate.

TODO

• Kubernetes Job configurations, e.g. backoffLimit, completions, ... etc.
• Documentations (including how to delete it if it fails)
• E2E tests
• Improve observability
• Test if the Job fails to connect to Redis.

Related issue number

Closes #1286

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

# Step 0: Create a Kubernetes cluster
# Step 1: Install a KubeRay operator with this PR
# Step 2: Create a GCS FT-enabled 
# (path: ray-operator/config/samples)
kubectl apply -f ray-cluster.external-redis.yaml

# Step 3: Wait until head Pod is running and ready.

# Step 4: Log in to the Redis Pod
kubectl exec -it $REDIS_POD -- bash

# Step 5: Check Redis data (in the Redis Pod)
redis-cli -a "5241590000000000"
KEYS *
# [Example output]: 1) "4c31680c-2a27-4788-92d8-6867188ecdd8"
HGETALL  "4c31680c-2a27-4788-92d8-6867188ecdd8"

# [Example output]
# 57) "4c31680c-2a27-4788-92d8-6867188ecdd8@KV:@namespace_dashboard:DASHBOARD_AGENT_PORT_PREFIX:19ca12675b387fb9e69ba6613b6c47ebc38ce441176b931fb052e09e"
# 58) "[52365, 43258]"
# 59) "4c31680c-2a27-4788-92d8-6867188ecdd8@KV:@namespace_usage_stats:extra_usage_tag_dashboard_used"
# 60) "False"

# Step 6: Delete the RayCluster
kubectl delete rayclusters.ray.io raycluster-external-redis 

# Step 7: Repeat Step 4 and Step 5
KEYS *
# [Expected output]: (empty list or set)

[kevin85421]

cc @smit-kiri @JoshKarpel

[architkulkarni]

Looks good overall, just a few minor questions. Feel free to address them in a followup PR if needed

[architkulkarni]

Why don't we need this check anymore? Won't we get an error if Annotations is nil?

[kevin85421]

The nil map in Golang can be a bit tricky. It won't throw an exception if you look up a nil map, but it will throw an exception when you attempt to write to a nil map.

A nil map behaves like an empty map when reading, but attempts to write to a nil map will cause a runtime panic;

• ref: https://go.dev/blog/maps

• example:

  package main
  
  import "fmt"
  
  func main() {
     var rect map[string]int
     val, ok := rect["hi"] 
     fmt.Println(val, ok) // 0, false
     rect["height"] = 10 // panic: assignment to entry in nil map
     fmt.Println(rect["height"])
  }

[architkulkarni]

It's unfortunate that we have to use a private API for this. Do you know if we test this codepath in the Kuberay repo against the Ray nightly wheels? That will help us catch the error if the private function changes its behavior or name

[kevin85421]

Open a good first issue.
#1422

[architkulkarni]

Should we do something if the status is Failed? (Maybe log a message at least?)

[kevin85421]

Updated aca1b0a