[tune] PBT exploits trials that have no checkpoint, yet

[krfricke]

It seems that PBT sometimes tries to exploit trials that are currently being restarted or have just been restarted with old statistics.

PBT tries to exploit trials that never checkpointed. While this could be acceptable for hyperparameter exploitation, it will not progress training.

We should investigate this to see if this is actually the case.

On a different note, PBT pauses function trainable trials that never checkpointed and tries to resume them later. This doesn't work as there is no checkpoint to resume from.

[xwjiang2010]

I don't quite get this one, can you elaborate?

[krfricke]

I've looked into this and updated the issue description and title.

I think one of the main problems is that pbt both exploits and tries to pause/resume trials for which it has not received a checkpoint. These trials will always run from scratch.

The solution for the horovod test is to either checkpoint every step or only report once at the end of the iteration - I'll open a PR for that

[krfricke]

@justinvyu has this been resolved by #28509?

[justinvyu]

I believe that PR doesn't fix this issue. My understanding of this issue is that checkpoint_interval > perturbation_interval, which is part of the user-defined configuration. I don't have a good way of fixing the case where the user doesn't checkpoint in time for the first perturbation. I have added suggestions in updated examples/guides that recommends checkpointing on every perturbation, or more often than perturbations (ex: this guide). Perhaps we should log a warning and recommend a different checkpointing interval. However, one problem with function checkpointing is that the interval is not controlled by Tune.

I do have a PR that I'll post today that fixes a related issue, where PBT only pulls the checkpoint of trials in the upper quantile, but it's possible for a lower-quantile trial to get upgraded. In this case, my fix was to always save the checkpoint of each trial, since there's no guarantees of which trial will need to be exploited in the next perturbation round.

[krfricke]

@justinvyu with your improvements to PBT, this should be resolved, right?

[krfricke]

Closing this for now