[tune integration] Don't process nan/inf results #7
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Justin Yu <[email protected]>
justinvyu
changed the title
Signed-off-by: Justin Yu <[email protected]>
7 tasks
Signed-off-by: Justin Yu <[email protected]>
krfricke
pushed a commit
to ray-project/ray
that referenced
this pull request
Jan 9, 2023
Skips a zoopt searcher test that's causing the `test_searchers` suite in CI to be flaky. Skipping as this is not a Tune issue and needs to be fixed in the zoopt library. I re-enabled this test in #31147 since I thought that nan/inf error handling had been fixed in a recent zoopt release. However, nan/inf values will still cause an error if reported on trial complete. This test should be enabled after polixir/ZOOpt#7 is included in the next zoopt release. Signed-off-by: Justin Yu <[email protected]>
AmeerHajAli
pushed a commit
to ray-project/ray
that referenced
this pull request
Jan 12, 2023
Skips a zoopt searcher test that's causing the `test_searchers` suite in CI to be flaky. Skipping as this is not a Tune issue and needs to be fixed in the zoopt library. I re-enabled this test in #31147 since I thought that nan/inf error handling had been fixed in a recent zoopt release. However, nan/inf values will still cause an error if reported on trial complete. This test should be enabled after polixir/ZOOpt#7 is included in the next zoopt release. Signed-off-by: Justin Yu <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Justin Yu [email protected]
Currently,
nanresults causes the binary search happening insideSRacosTune.update_classifierto cause a stack overflow error from continuing to search recursively. This is becausex > nanandx < nanis always false. This PR adds a check for invalid solution results (including nans and infs) when completing a trial.One question: should
+/- infresults be processed? Tune usually ignores these along with nans for other search algorithms.