Move Jenatton test function to appropriate file #2679
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D61431983 |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2679 +/- ##
==========================================
- Coverage 95.28% 95.26% -0.02%
==========================================
Files 495 495
Lines 47800 47865 +65
==========================================
+ Hits 45545 45600 +55
- Misses 2255 2265 +10 ☔ View full report in Codecov by Sentry. |
esantorella
added a commit
to esantorella/Ax
that referenced
this pull request
Aug 20, 2024
Summary: Pull Request resolved: facebook#2679 I didn't do this in the previous diff so that it would be easier to review. Reviewed By: Balandat Differential Revision: D61431983
|
This pull request was exported from Phabricator. Differential Revision: D61431983 |
esantorella
force-pushed
the
export-D61431983
branch
from
6087a3a
to
34f5802
Compare
…book#2674) Summary: Pull Request resolved: facebook#2674 Context: This is an alternative to D61431979. Note: There are benchmarks that do not use `BenchmarkRunner`, but I plan to have them all use `BenchmarkRunner` in the future. `BenchmarkRunner` technically supports benchmarks without a ground truth, but that functionality is never used, and there aren't any Ax benchmarks that are noisy *and* don't have a ground truth. It is not conceptually clear how such a case should be benchmarked, so it is better to not over-engineer for that need, which may never arise. Instead, benchmarks that lack a ground truth but are deterministic can be treated as noiseless problems with a ground truth, and we can reap support for problems without a ground truth. Also, `BenchmarkRunner` has some methods that must either be defined or not defined depending on whether there is a ground truth. They can't be abstract because they will not always be defined. With this change, we can make the ground-truth methods abstract and get rid of the rest. This PR: - Rewrites docstrings - Removes method `get_Y_Ystd` - Makes `get_Y_true` and other methods abstract - Removes functionality for the case where `get_Y_true` raises a `NotImplementedError` Reviewed By: ItsMrLin Differential Revision: D61483962
Summary: Pull Request resolved: facebook#2675 Context: In a future refactor that will enable more flexible and powerful best-point functionality, every BenchmarkProblem's runner will be able to produce an "oracle" value (possibly the ground truth) for any arm, in-sample or not, with a function like `BenchmarkRunner.evaluate_oracle(arm=arm)`, with the problem handling computation and the runner formatting results. However, the current `BenchmarkRunner` and `BenchmarkMetric` setup currently doesn't cover every benchmark. Consolidating on `BenchmarkRunner` and `BenchmarkMetric` will enable the refactor, make it easier to universalize functionality like handling of constraints, noise, and inference regret, and will also allow for deleting some LOC for more custom problems. Current `BenchmarkRunner`s only handle problems that can consume tensor-valued arguments: BoTorch synthetic problems and surrogate problems. This isn't a good fit for problems like Jenatton that have a hierarchical search space and can have some parameters not passed. Because Ax always passes parameters and only sometimes represents them as tensors, a `TParameterization` is a more natural abstraction to handle parameters than a tensor. This PR: - Introduces `ParamBasedTestProblem`, which is like a BoTorch synthetic test problem but consumes a `TParameterization` rather than a tensor - Added `ParamBasedProblemRunner`, which shares a base class `SyntheticProblemRunner` and most functionality with `BotorchTestProblemRunner` (so it is a `BenchmarkRunner` and supports both observed and unboserved noise). Differential Revision: D60996475
…#2676) Summary: Pull Request resolved: facebook#2676 This PR: - Has Jenatton use `ParamBasedTestProblem` so that it can use `ParamBasedProblemRunner`, and also have it use `BenchmarkMetric`; get rid of specialized Jenatton runners and metrics. This enables Jenatton to handle noisy problems, whether noise levels are observed or not, like other benchmark problems, and will make it easy to add constraints or benefit from other new functionality. - Does *not* clean up the now-unnecessary Jennaton metric file; that happens in the next diff. Differential Revision: D61502458 Reviewed By: Balandat
Summary: Pull Request resolved: facebook#2679 I didn't do this in the previous diff so that it would be easier to review. Reviewed By: Balandat Differential Revision: D61431983
|
This pull request was exported from Phabricator. Differential Revision: D61431983 |
esantorella
force-pushed
the
export-D61431983
branch
from
34f5802
to
fa2ad72
Compare
|
This pull request has been merged in b9dd1b0. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary: I didn't do this in the previous diff so that it would be easier to review.
Reviewed By: Balandat
Differential Revision: D61431983