[AIR output] Warnings for AIR_VERBOSITY is confusing

[scottsun94]

What happened + What you expected to happen

Confusing instructions. Not sure what the second half of it means.

2023-03-28 09:36:07,926 WARNING tune.py:526 -- Testing new AIR console output flow with verbosity=1. This will also disable the old flow - setting it to 0 now.

Can we change it to something like
2023-03-28 09:36:07,926 WARNING tune.py:526 -- AIR_VERBOSITY is set to 1 and new AIR console output is enabled. To see the old console output, don't use or unset AIR_VERBOSITY.

Versions / Dependencies

Nightly

Reproduction script

n/a

Issue Severity

Low: It annoys or frustrates me.

[xwjiang2010]

• time_this_iter vs time_total_s
time_total_s means "Accumulated time in seconds for this entire trial."
  time_this_iter is just for this past iteration.

• "running for 00:00:18:09" captures the entire running time (from the beginning when you say xx.fit()).

• You should see checkpoint information, if it is saved. But whether to save it is a user call. Can you first make sure that checkpoint saving is turned on for every iteration? If it's off, we won't print that line for the iteration.

[xwjiang2010]

• "Configurations for TorchTrainer: "
  what should we say if there is no specified configuration for starting the trainer (a lot of times the case)?

Right now, I make it
"started with configuration {...}" v.s. "started".

[scottsun94]

• time_this_iter vs time_total_s
time_total_s means "Accumulated time in seconds for this entire trial."
  time_this_iter is just for this past iteration.
• "running for 00:00:18:09" captures the entire running time (from the beginning when you say xx.fit()).
• You should see checkpoint information, if it is saved. But whether to save it is a user call. Can you first make sure that checkpoint saving is turned on for every iteration? If it's off, we won't print that line for the iteration.

Got it. Thanks! We can do something like this then?

• Training time of this iteration == time_this_iter
• Total running time == time_total_s at the end of last iteration
• Total training time == sum of all iterations' time_this_iter?

Iteration 4 finished at 2023-03-28 10:09:42. Training time of this iteration: 8.96s
-------------------------
loss             2.24726
-------------------------
Checkpoint saved at: mnt/ray_result/...

Training finished (4 iterations) at 2023-02-24 12:35:39. Total training time: 39.68s. Total running time: 46.97s
Last checkpoint saved at: mnt/ray_result/...