[Dashboard] Increase the RPC timeout for the snapshot API

[rkooo567]

Why are these changes needed?

The current RPC timeout is too short (2s), and we've discovered Ray components not responding within the current timeout range occasionally under pressure. This is something we will fix, but it's better if we can have a longer timeout. The current timeout is configured as 2X of the polling frequency.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[wuisawesome]

Can we make this configurable via env var?

[wuisawesome]

On a somewhat related note, I wonder if we should distinguish between timeouts and other error types...

[galenhwang]

Do we want to pass timeout in as a query param for any of these? Similar to the component_activities route.

ray/dashboard/modules/snapshot/snapshot_head.py

Lines 132 to 136 in de2db06

	self.get_job_info(),
	self.get_job_submission_info(),
	self.get_actor_info(actor_limit),
	self.get_serve_info(),
	self.get_session_name(),

That way, if any clusters have long snapshots, we could use a feature flag to target specific clusters to extend the timeout period.

[jjyao]

I may miss some context here: will snapshot API be replaced completely by component_activities. Anything we do for snapshot, should we consider it for component_activities as well?

[wuisawesome]

Actually, also are we sure we need this in light of #27589?

[rkooo567]

@wuisawesome it is still safe to do this. I think 2 seconds are too drastic timeout in general

[rkooo567]

@jjyao let me double check if this change is reflected to the component activities too.

[rkooo567]

@wuisawesome @galenhwang I will go with Galen's suggestion for the timeout configuration (it can be specified via Http req argument)

[galenhwang]

Would we able to backport this to previous Ray versions? This mainly affects pre-2.0 versions.

[rkooo567]

cc @matthewdeng to answer #28330 (comment)

[rkooo567]

@jjyao confirms it also affects timeout for component_activities . I think snapshot is still used along with component_activities

[matthewdeng]

We could perform a patch release for older versions (perhaps just 1.13), but would be hesitant to do so unless there is a pressing need.

[rkooo567]

Added the timeout support. I will try merging it once the CI passes.

[rkooo567]

cc @alanwguo I need the code owner approval too

Addressed