[core][state] Task events backend - Port state API [5/n]

[rickyyx]

Signed-off-by: rickyyx [email protected]

Why are these changes needed?

Previous PRs:

• [core][state] Task events backend: config and interface definitions [0/n]  #30829:
• [core][state] Task events backend - split drop count on worker #30953:
• [core][state] Task events backend - worker task event buffer implementation [1/n] #30867:
• [core][state] Task events backend - gcs update path and interface [2/n] #30979:
• [core][state] Task events backend - gcs read path [3/n] #30934
• [core][state] Task events backend - port profile events and turn on task backend [4/n]  #31207

This PR:

With the change,

• list_tasks now will return tasks with attempt number as an additional column.
• get_task might return multiple task attempt entries if there are retries.

There is also some plumbing in the test and in core (esp in the test logic) given the changes. Major changes in the PR are:

• Add limit support to GcsTaskManager
• Change the state aggregator to get tasks from GCS.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[ericl]

Could we split out the unrelated changes to TaskManager? These are high-risk changes, so we should review them independently of the status API.