[Core, Observabilty] Create metric for running ray jobs

[alanwguo]

Tracking the running ray jobs would be useful. Alerts could be created on if ray clusters are left alive without a running job.

[ekdnam]

Hi @alanwguo @jjyao ! I would like to work on this. Can you please let me know from where should I start?

[alanwguo]

@jjyao can we actually have someone from ray core offer guidance here? I'm not familiar with the core code that emits metrics.

This will likely require a C++ change, @ekdnam , if you're comfortable

[jjyao]

@dentiny do you want to take this one? Check metrics for actors as a starting point. The code is in gcs_actor_manager.cc:

ray::stats::STATS_actors.Record(
            num_actors,
            {{"State", rpc::ActorTableData::ActorState_Name(key.first)},
             {"Name", key.second},
             {"Source", "gcs"},
             {"JobId", ""}});

For jobs, the code will be in gcs_job_manager.cc, we want a ray_jobs gauge metric with a State label (possible values are: RUNNING and FINISHED).

[dentiny]

Thanks @jjyao! I would love to take a look.

[ekdnam]

Hi @alanwguo ! Sorry for not taking this up, I have been caught up in some personal stuff for the past few weeks.