Multi & single deployment large scale test

[jiaodong]

Summary

• Added multi deployment test with 1k replica and 100 deployments
• Multi deployment will randomly pick next endpoint from all 100 existing ones, include self, and stop at depth of 5
• Warmup at each endpoint / deployment has its own ray remote task and executed in parallel
• Each node has its own HTTP Proxy as well as wrk job simulating load balanced traffic
• Metrics are parsed and aggregated as one, and shows per thread/node/cluster level
• A bunch of refactoring

Test plan

1 deployment, 1k replicas, 10 cpu per node, 100 connections per wrk:

https://beta.anyscale.com/o/anyscale-internal/projects/prj_UgMsAz5Rqk2iuvHmyrBH3S4E/clusters/ses_TQztbsGqe6rauG23JdaJihgi?command-history-section=command_history

100 deployments, 1k replicas, 10 cpu per node, 100 connections per wrk:

https://beta.anyscale.com/o/anyscale-internal/projects/prj_UgMsAz5Rqk2iuvHmyrBH3S4E/clusters/ses_89uwNQunRpcDsiBZ73y8BRYi

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[edoakes]

@jiaodong please fix merge conflicts from the PR I just merged!

[edoakes]

Code looks good. From the results it looks like we have some work to do in tuning & understanding the perf.