[Feature] Improve and fix Prometheus & Grafana integrations

[kevin85421]

Why are these changes needed?

The old document (observability.md) of Prometheus and Grafana has some issues that make users hard to follow.

• Inconsistency between install.sh and observability.md

  • ServiceMonitor example in the doc and ServiceMonitor instance created by the script.
  • PodMonitor example in the doc and PodMonitor instance created by the script.

• Inconsistency between observability.md and KubeRay operator kuberay/#230.

  • The document uses port 9001 as a metrics endpoint, but KubeRay operator uses 8080 as a default metrics endpoint.

• Some missing pieces:

  • Both examples of ServiceMonitor and PodMonitor in observability.md lack some labels to match serviceMonitorSelector and podMonitorSelector.
  • How to access Prometheus UI?
  • How to access Grafana UI?
  • How to import Ray Dashboard into Grafana?

Related issue number

Closes #871

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

• Prometheus UI: Prometheus can detect both ServiceMonitor (ray-head-monitor) and PodMonitor (ray-worker-monitor).

• Grafana UI: Load Ray Dashboard into Grafana via config/grafana/dashboard_default.json.

[kevin85421]

cc @Yicheng-Lu-llll would you mind reviewing this PR? Thanks!

[kevin85421]

Gentle ping @scarlet25151. Would you mind taking a look at this PR? Thank you!

[architkulkarni]

Got this error at this step:

Unable to listen on port 3000: Listeners failed to create with the following errors: [unable to create listener: Error listen tcp4 0.0.0.0:3000: bind: address already in use]
error: unable to listen on any of the requested ports: [{3000 3000}]

But I'm not sure what could be using port 3000. Do I need to cancel one of the previous steps?

[architkulkarni]

Looks like I already have grafana running for some reason, I think it's unrelated to this tutorial:

sudo lsof -i :3000
Password:
COMMAND    PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
grafana-s 3915 archit   21u  IPv6 0xd6d807306e5fa78f      0t0  TCP *:hbci (LISTEN)

Let me figure out how to stop it. I don't know if it's a common enough situation to mention in this document.

[architkulkarni]

Naively killed the process but it kept restarting. The solution here https://unix.stackexchange.com/questions/601063/port-3000-is-always-being-hogged-by-grafana-server
is to run brew services stop grafana. Probably don't need to mention in this doc

[architkulkarni]

Went through the tutorial, it all works! Left some minor suggestions.