[tune] deflake pbt. #21366
Merged
[tune] deflake pbt. #21366
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
krfricke
approved these changes
Jan 4, 2022
Comment on lines
121
to
125
| self._cur_order += 1 | ||
| checkpoint.order = self._cur_order | ||
| self.on_checkpoint_internal(checkpoint) | ||
|
|
||
| def on_checkpoint_internal(self, checkpoint): |
There was a problem hiding this comment.
Do we need the on_checkpoint_internal function here? Seems like we never call it anywhere else
There was a problem hiding this comment.
oh right, making it private
There was a problem hiding this comment.
What's the purpose of splitting out on_checkpoint_internal into a separate method altogether?
What logic should be contained in the external method vs. the internal method?
There was a problem hiding this comment.
Yep, would also vote to remove the function completely and just keep on_checkpoint
There was a problem hiding this comment.
sure. combining the two.
|
To fix the build errors, merge current master |
krfricke
approved these changes
Jan 4, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
We use
trial.checkpointto restore a perturbed trial. Currently trial.checkpoint is looking at both in-memory and persistent checkpoints to find the most recent one. The definition of "the most recent one" is based on iteration. This may no longer be a valid assumption in PBT case, consideringtrial_low_quantilemay have an iter=2_persistent_checkpoint as well as a iter=1_in_memory_checkpoint (perturbed fromtrial_upper_quantile).BTW, I think there are a few other issues to clean up for PBT case, depending on how important we think PBT is. For example,
Related issue number
Checks
scripts/format.shto lint the changes in this PR.