[tune] Add test for heterogeneous resource request deadlocks

[krfricke]

Why are these changes needed?

This adds a test for potential resource deadlocks in experiments with heterogeneous PGFs. If the PGF of a later trial becomes ready before that of a previous trial, we could run into a deadlock. This is currently avoided, but untested, flagging the code path for removal in #21387.

Related issue number

Closes #21387

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[xwjiang2010]

Thanks Kai. This is neat.

In the long term, we may want choose_trial_to_run(live_trials_of_pgf) like we discussed. has_resources_for_trial is not functioning any more.