Discussions about which metrics we need to stat

[jovany-wang]

We have integrated opencensus library into raylet and added 2 example metrics: CurrentWorker and TaskReceivedCount.

I think this issue can be used to collect our requirements about metrics. I racked my brains to think of these metrics:

• RedisLatency: The latency of a redis operator. (finished at Integrate metric items into raylet #4602)
• CompletedTaskCount: How many completed tasks.
• TaskElapsedTime: The time that a task elapsed in backend.
• ObjectCount: How many objects we are holding.

I discussed with @raulchen about TaskElapsedTime offline, and we thought it's unnecessary to add this metric because the change is a bit huge, we should add a start_time for TaskInfo, and this should be implemented by tracing system which can trace the lifecycle of tasks, actor, etc.

So, what other metrics need to be stated?

[jovany-wang]

@raulchen @guoyuhong
@ericl @pcmoritz @robertnishihara

[ericl]

I would probably also add the metrics emitted in debug_state.txt as well, which should cover internal data structure sizes and things like that.

Some metrics on what the raylet is spending time doing would also be useful, i.e., time making scheduling decisions.

[robertnishihara]

@jovany-wang One thing that would be incredibly useful is to know how long the raylet spends in each event handler? E.g., what is the maximum amount (and the whole distribution ideally) of time it spends responding to a task submission or to a object becoming available.

[jovany-wang]

@robertnishihara Thank you for reminding me of this.

[rkooo567]

Added new stats.