Run ModelLauncher with a BenchmarkRunner and BenchmarkMetric #2681
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D61397457 |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2681 +/- ##
==========================================
- Coverage 95.28% 95.26% -0.02%
==========================================
Files 495 495
Lines 47800 47867 +67
==========================================
+ Hits 45545 45600 +55
- Misses 2255 2267 +12 ☔ View full report in Codecov by Sentry. |
esantorella
added a commit
to esantorella/Ax
that referenced
this pull request
Aug 20, 2024
…k#2681) Summary: Pull Request resolved: facebook#2681 Similar to how Jennaton was migrated in D60996475, this change switches the ModelLauncher problem to use `BenchmarkRunner` and `BenchmarkMetric`. Differential Revision: D61397457 Reviewed By: Balandat
esantorella
added a commit
to esantorella/Ax
that referenced
this pull request
Aug 20, 2024
…k#2681) Summary: Pull Request resolved: facebook#2681 Similar to how Jennaton was migrated in D60996475, this change switches the ModelLauncher problem to use `BenchmarkRunner` and `BenchmarkMetric`. Differential Revision: D61397457 Reviewed By: Balandat
|
This pull request was exported from Phabricator. Differential Revision: D61397457 |
esantorella
force-pushed
the
export-D61397457
branch
from
8c214e2
to
ac547ac
Compare
esantorella
added a commit
to esantorella/Ax
that referenced
this pull request
Aug 20, 2024
…k#2681) Summary: Pull Request resolved: facebook#2681 Similar to how Jennaton was migrated in D60996475, this change switches the ModelLauncher problem to use `BenchmarkRunner` and `BenchmarkMetric`. Reviewed By: Balandat Differential Revision: D61397457
…book#2674) Summary: Pull Request resolved: facebook#2674 Context: This is an alternative to D61431979. Note: There are benchmarks that do not use `BenchmarkRunner`, but I plan to have them all use `BenchmarkRunner` in the future. `BenchmarkRunner` technically supports benchmarks without a ground truth, but that functionality is never used, and there aren't any Ax benchmarks that are noisy *and* don't have a ground truth. It is not conceptually clear how such a case should be benchmarked, so it is better to not over-engineer for that need, which may never arise. Instead, benchmarks that lack a ground truth but are deterministic can be treated as noiseless problems with a ground truth, and we can reap support for problems without a ground truth. Also, `BenchmarkRunner` has some methods that must either be defined or not defined depending on whether there is a ground truth. They can't be abstract because they will not always be defined. With this change, we can make the ground-truth methods abstract and get rid of the rest. This PR: - Rewrites docstrings - Removes method `get_Y_Ystd` - Makes `get_Y_true` and other methods abstract - Removes functionality for the case where `get_Y_true` raises a `NotImplementedError` Reviewed By: ItsMrLin Differential Revision: D61483962
Summary: Pull Request resolved: facebook#2675 Context: In a future refactor that will enable more flexible and powerful best-point functionality, every BenchmarkProblem's runner will be able to produce an "oracle" value (possibly the ground truth) for any arm, in-sample or not, with a function like `BenchmarkRunner.evaluate_oracle(arm=arm)`, with the problem handling computation and the runner formatting results. However, the current `BenchmarkRunner` and `BenchmarkMetric` setup currently doesn't cover every benchmark. Consolidating on `BenchmarkRunner` and `BenchmarkMetric` will enable the refactor, make it easier to universalize functionality like handling of constraints, noise, and inference regret, and will also allow for deleting some LOC for more custom problems. Current `BenchmarkRunner`s only handle problems that can consume tensor-valued arguments: BoTorch synthetic problems and surrogate problems. This isn't a good fit for problems like Jenatton that have a hierarchical search space and can have some parameters not passed. Because Ax always passes parameters and only sometimes represents them as tensors, a `TParameterization` is a more natural abstraction to handle parameters than a tensor. This PR: - Introduces `ParamBasedTestProblem`, which is like a BoTorch synthetic test problem but consumes a `TParameterization` rather than a tensor - Added `ParamBasedProblemRunner`, which shares a base class `SyntheticProblemRunner` and most functionality with `BotorchTestProblemRunner` (so it is a `BenchmarkRunner` and supports both observed and unboserved noise). Differential Revision: D60996475
…#2676) Summary: Pull Request resolved: facebook#2676 This PR: - Has Jenatton use `ParamBasedTestProblem` so that it can use `ParamBasedProblemRunner`, and also have it use `BenchmarkMetric`; get rid of specialized Jenatton runners and metrics. This enables Jenatton to handle noisy problems, whether noise levels are observed or not, like other benchmark problems, and will make it easy to add constraints or benefit from other new functionality. - Does *not* clean up the now-unnecessary Jennaton metric file; that happens in the next diff. Differential Revision: D61502458 Reviewed By: Balandat
Summary: I didn't do this in the previous diff so that it would be easier to review. Differential Revision: D61431983
…k#2681) Summary: Pull Request resolved: facebook#2681 Similar to how Jennaton was migrated in D60996475, this change switches the ModelLauncher problem to use `BenchmarkRunner` and `BenchmarkMetric`. Reviewed By: Balandat Differential Revision: D61397457
|
This pull request was exported from Phabricator. Differential Revision: D61397457 |
esantorella
force-pushed
the
export-D61397457
branch
from
ac547ac
to
f4bc194
Compare
esantorella
added a commit
to esantorella/Ax
that referenced
this pull request
Aug 21, 2024
…k#2681) Summary: Pull Request resolved: facebook#2681 Similar to how Jennaton was migrated in D60996475, this change switches the ModelLauncher problem to use `BenchmarkRunner` and `BenchmarkMetric`. Differential Revision: D61397457 Reviewed By: Balandat
esantorella
added a commit
to esantorella/Ax
that referenced
this pull request
Aug 21, 2024
…k#2681) Summary: Pull Request resolved: facebook#2681 Similar to how Jennaton was migrated in D60996475, this change switches the ModelLauncher problem to use `BenchmarkRunner` and `BenchmarkMetric`. Differential Revision: D61397457 Reviewed By: Balandat
|
This pull request has been merged in 9a83f87. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary: Similar to how Jennaton was migrated in D60996475, this change switches the ModelLauncher problem to use
BenchmarkRunnerandBenchmarkMetric.Differential Revision: D61397457