[Release Process] test oncall needs to fix issues on master branch earlier

[can-anyscale]

What happened + What you expected to happen

Tests that has been failing for too long without fixing took a long time to debug.

In particular, during 2.5.0 release, microbenchmark_38 failed for close to a month up to the release (#34915). During the release, it took about 10 days to resolve.

On previous release, Cuong played the role of test-cop to open issues and pushing people to fix tests. Cuong did that for one particular OSS team and that has been working pretty well.

For the next releases, we will expand this test policy, test cop idea to all teams within OSS. The test cop is also now played mostly as a bot.

Versions / Dependencies

N/A

Reproduction script

N/A

Issue Severity

None

[stale]

Hi, I'm a bot from the Ray team :)

To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.

If there is no further activity in the 14 days, the issue will be closed!

• If you'd like to keep the issue open, just leave any comment, and the stale label will be removed!
• If you'd like to get more attention to the issue, please tag one of Ray's contributors.

You can always ask for help on our discussion forum or Ray's public slack channel.