[Tune] Tuner.restore from a local directory that has moved

[justinvyu]

Why are these changes needed?

Problem

Upon restore, parts of the Tuner state are not updated, including local_dir, experiment name, and when trials get recreated from the TrialRunner experiment checkpoint, the paths stored inside the the trial checkpoints are relative to the old experiment directory. See the issues linked below for more context.

Solution

1. For local checkpointing, Tuner.restore(new_experiment_path) now properly sets the new local_dir and experiment name for restored trials based on the path that is passed in.
   • Example: If the experiment directory moves from /original/path/exp_name/ to /moved/path/new_exp_name/, /moved/path is the new local_dir, and new_exp_name is the new experiment name.
2. Trial restoration now sets the checkpoints stored in the trial CheckpointManager to the correct paths based on what's found in the trial dir.
3. Other trial state such as pickled_error_file and error_file are now stored as relative paths with respect to the trial logdir. (They used to be saved/loaded as absolute paths.)
4. Added tests for this workflow (both through Tuner.restore and tune.run(resume=...)). Also, added a test that shows loading a ResultGrid via Tuner.restore().get_results() and accessing checkpoints works even if the directory has moved.

Notes for future

This PR changes TrialRunner.resume functionality to reconstruct the trial logdir as a subdirectory of the local_checkpoint_dir, which is the experiment directory. The checkpoint paths are then updated to be relative to the trial logdir. In the future, unit tests that restore trial checkpoints should set:

experiment_dir = ... # some temp dir
trial = Trial(local_dir=experiment_dir)
trial_runner = TrialRunner(local_checkpoint_dir=experiment_dir)

This matches the experiment directory structure that a normal Tune run assumes.

experiment-directory/
    experiment-checkpoint.json
    trial-directory/
        checkpoint_00000/

Related issue number

Closes #28082
Closes #28621

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[krfricke]

Looks great so far!

[Yard1]

Looks awesome!

[justinvyu]

I needed to set local_dir to be the same as TrialRunner._local_checkpoint_dir for these tests to work. This is because TrialRunner.resume will set the trial checkpoint paths to be relative to its _local_checkpoint_dir and assumes a directory structure of:

<experiment logdir> == _local_checkpoint_dir
    | experiment-checkpoint.json
    | <trial.logdir>
        | checkpoint_00000

Previously, the trial checkpoints and experiment checkpoints in the test were being saved in two separate places. On resume, the trial checkpoint dirs get updated and point at the wrong location.

<experiment logdir> == _local_checkpoint_dir
    | experiment-checkpoint.json

<default trial logdir>
    | checkpoint_00000

Is this trial directory structure okay to assume? Wondering if it will be too easy for future tests to make this mistake.

[Yard1]

I think it's fine to hardcode it provided there's a comment explaining it.

[Yard1]

LGTM, let's wait for CI to pass

[amogkam]

@justinvyu can you merge master again please