[air/output] Improve leaked mentions of Tune concepts

[krfricke]

Why are these changes needed?

Ray Tune is the execution backend for Ray Train. This means that sometimes error/warning messages use Tune concepts, that don't make sense in a single-trial run, such as with Ray Train trainers.

This PR improves three such occurrences:

1. The insufficient resources warnings message has been adjusted in the case where only one trial is run
2. Calculation of max_pending_trials now uses search_alg.total_samples as the minimum, which was an oversight before.
3. On interrupt of a training run, a Tuner.restore() message was suggested, but it should be Trainer.restore()

With these fixes, we will see message such as:

Insufficient resources

Training has not started in the last 30 seconds.
This could be due to the cluster not having enough resources available.
You asked for 4 CPUs and 2 GPUs, but the cluster only 
has 4 CPUs and 1 GPUs available.
Stop the training and adjust the required resources (e.g. via the 
`ScalingConfig` or `resources_per_trial`, or `num_workers` for rllib), 
or add more resources to your cluster.

Maximum number of pending trials

The message

2023-03-28 13:51:14,756 WARNING trial_runner.py:1576 -- The maximum number of pending trials has been automatically set to the number of available cluster CPUs, which is high (176 CPUs/pending trials). If you're running an experiment with a large number of trials, this could lead to scheduling overhead. In this case, consider setting the `TUNE_MAX_PENDING_TRIALS_PG` environment variable to the desired maximum number of concurrent trials.

will not turn up for Ray Train runs anymore.

Restore

For Train runs, will now be:

Experiment has been interrupted, but the most recent state was saved.
Continue running this experiment with: Trainer.restore(path="/Users/kai/ray_results/debug_execution_restart_dpt7", ...)

We can't get rid of all tune-related concepts just yet, but this is at least an improvement.

Related issue number

Closes #33839 (or parts of it)

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[gjoliver]

I like the error message refactoring.
want to discuss about the long term plan of how to differentiate between trainer and tuner though.

[gjoliver]

@justinvyu can we have a chat about adding parameters like _trainer_api in Tuner :)

[krfricke]

I think we generally need a good idea how to pass this information.

To me it feels like there should be some time of context. We have different requirements for different ML jobs. Even rllib vs Train has different requirements (e.g. default metrics to show), and maybe even rllib's single algorithms.

We don't have that story, yet, so in order to unblock this work, I think we can go ahead with the private flags. But yes, we should resolve this (also for telemetry).

[gjoliver]

explanation of what this is?
can we compute whether there is only a single Trial ourselves?
I mean, feel like it would be nice to be able to avoid such a trainer parameter on Tuner init.

[krfricke]

I've tried this before, and long story short, it's not very straightforward due to the fact that we need some of the information pretty early, but number of trials is only calculated later. It also can lead to confusing situations - e.g. it's totally valid to use Tuner(trainable, tune_config=TuneConfig(num_samples=1)) for iteration and still expect to see a Tuner.restore() hint at the end. Tracking which object actually was the entrypoint saves us from those problems.

[gjoliver]

ok, I just feel like passing such a generic sounding parameter all the way through so many components like Tuner, BackendExecutor, etc just to be able to show the right output message is kind of too heavy.
we should probably have a way out of this if we want to live with it for now. @justinvyu
another idea is maybe passing this bit through a LoggingConfig or something? we probably need such a config class anyways?

[krfricke]

We're doing the same at the moment with _tuner_api, so in that sense it's consistent :-D

This is not a logging configuration in my opinion. Users should not "configure" which output/error messages they want to see. It's more of a runtime context.

Ray core has a runtime context object, I think we just need something similar for Ray AIR.

[scottsun94]

Restore

For Train runs, will now be:> Restore

For Train runs, will now be:

Experiment has been interrupted, but the most recent state was saved.
Continue running this experiment with: Trainer.restore(path="/Users/kai/ray_results/debug_execution_restart_dpt7", ...)

Is Experiment a concept for Train users? Should it just be

Training has been interrupted, but the most recent state was saved.
 Resume training with: Trainer.restore(path="/Users/kai/ray_results/debug_execution_restart_dpt7", ...)

Other LGTM. Thanks!

Experiment has been interrupted, but the most recent state was saved.
Continue running this experiment with: Trainer.restore(path="/Users/kai/ray_results/debug_execution_restart_dpt7", ...)

Is Experiment a concept for Train users? Should it just be

**Training** has been interrupted, but the most recent state was saved.
 **Resume training** with: Trainer.restore(path="/Users/kai/ray_results/debug_execution_restart_dpt7", ...)

Others LGTM. Thanks!

[krfricke]

Updated, thanks!

[gjoliver]

sorry I missed this.
let's add a TODO and unblock the output work.