[release][tune][build_base] Fix tune_scalability_durable_trainable configuration to force experiment syncs less often

[justinvyu]

Deflakes tune_scalability_durable_trainable release test by configuring it so that forced experiment checkpoint syncs happen less frequently, cutting down on the test runtime.

Why are these changes needed?

The tune_scalability_durable_trainable test had a performance regression (running for ~300s longer) and is flaky. This was due to experiment checkpoint syncing to cloud being forced more often due to a fix by #31131. Each forced sync causes the driver to wait until that sync is finished before launching a new one, which accounts for the drastically increased runtime. This PR fixes the issue by increasing the number of checkpoints to keep, which in turn reduces the frequency of forced cloud syncs.

Question: How can we provide a better default for users? Can we provide a maximum amount of time spent waiting on cloud syncs? So that even if the user configures keep_checkpoints_num in such a way that syncs are getting forced very frequently, their script runtime doesn't get affected past a certain amount.

• Tracking this here: [Tune] Excessive experiment state syncing can cause significant slowdown #31613, to be addressed in a followup.

Related issue number

Closes #31506

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(