[Dashboard] Revisit Reporter Agent communication protocol to use proto instead of JSON

[alexeykudinkin]

What happened + What you expected to happen

Currently, Reporter Agent reports resource utilization as JSON, which quickly becomes substantial overhead on the Dashboard process:

1. Volume of the data passed in scales w/ the # of workers and nodes
2. Dashboard is a single Python process (that now has to parse a lot of JSON)

#45048 partially alleviates the problem of avoiding blocking the event-loop while parsing the JSON, but it doesn't resolve the underlying problem of inefficiency JSON as a format for application with the large number of stats being passed around.

Instead, we should rebase this payload handling to be proper Protobuf.

Versions / Dependencies

2.20

Reproduction script

1. Launch a Ray cluster with 100 nodes and 1000 workers
2. Profile Dashboard process

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[rynewang]

Q: does this JSON parsing overhead really affect dashboard API latency to the extent of 10s of seconds? If it's already in another thread, the GIL is preempted to handle the main loop every a few hundred milliseconds so we should witness an increase of latency but not to multiple seconds. Do we have latency data after #45048 is applied?

[hongchaodeng]

Q: does this JSON parsing overhead really affect dashboard API latency to the extent of 10s of seconds?

+1
Let's try to reproduce this first. Then we can profile it.