Allow more flexible definition of which trial statuses to fit. #2432
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D56634321 |
bernardbeckerman
pushed a commit
to bernardbeckerman/Ax
that referenced
this pull request
May 6, 2024
…ook#2432) Summary: Changes `observations_from_data` and `observations_from_map_data` signatures, replacing arg `fit_abandoned: bool` -> `statuses_to_fit` and `statuses_to_fit_map_metric` which are both `Optional[List[TrialStatus]]`. In `observations_from_data`, these default to `None` in which case `{TrialStatus.COMPLETED}` is used for any MapMetrics, else `NON_ABANDONED_STATUSES`, i.e., all trial statuses except abandoned. In `observations_from_map_data`, `NON_ABANDONED_STATUSES` are the default for all metrics. NOTE: As of this diff, any rows of `Data.df` containing `metric_names` that don't exist on `experiment` will be filtered out ([pointer](https://www.internalfb.com/diff/D56634321?permalink=330145676765780)). Reviewed By: saitcakmak Differential Revision: D56634321
bernardbeckerman
force-pushed
the
export-D56634321
branch
from
8e26abd
to
760034a
Compare
|
This pull request was exported from Phabricator. Differential Revision: D56634321 |
bernardbeckerman
pushed a commit
to bernardbeckerman/Ax
that referenced
this pull request
May 6, 2024
…ook#2432) Summary: Changes `observations_from_data` and `observations_from_map_data` signatures, replacing arg `fit_abandoned: bool` -> `statuses_to_fit` and `statuses_to_fit_map_metric` which are both `Optional[List[TrialStatus]]`. In `observations_from_data`, these default to `None` in which case `{TrialStatus.COMPLETED}` is used for any MapMetrics, else `NON_ABANDONED_STATUSES`, i.e., all trial statuses except abandoned. In `observations_from_map_data`, `NON_ABANDONED_STATUSES` are the default for all metrics. NOTE: As of this diff, any rows of `Data.df` containing `metric_names` that don't exist on `experiment` will be filtered out ([pointer](https://www.internalfb.com/diff/D56634321?permalink=330145676765780)). Reviewed By: saitcakmak Differential Revision: D56634321
bernardbeckerman
force-pushed
the
export-D56634321
branch
from
760034a
to
f2b1dda
Compare
|
This pull request was exported from Phabricator. Differential Revision: D56634321 |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2432 +/- ##
=======================================
Coverage 95.30% 95.30%
=======================================
Files 494 494
Lines 47920 47990 +70
=======================================
+ Hits 45670 45739 +69
- Misses 2250 2251 +1 ☔ View full report in Codecov by Sentry. |
bernardbeckerman
pushed a commit
to bernardbeckerman/Ax
that referenced
this pull request
May 6, 2024
…ook#2432) Summary: Changes `observations_from_data` and `observations_from_map_data` signatures, replacing arg `fit_abandoned: bool` -> `statuses_to_fit` and `statuses_to_fit_map_metric` which are both `Optional[List[TrialStatus]]`. In `observations_from_data`, these default to `None` in which case `{TrialStatus.COMPLETED}` is used for any MapMetrics, else `NON_ABANDONED_STATUSES`, i.e., all trial statuses except abandoned. In `observations_from_map_data`, `NON_ABANDONED_STATUSES` are the default for all metrics. NOTE: As of this diff, any rows of `Data.df` containing `metric_names` that don't exist on `experiment` will be filtered out ([pointer](https://www.internalfb.com/diff/D56634321?permalink=330145676765780)). Reviewed By: saitcakmak Differential Revision: D56634321
bernardbeckerman
force-pushed
the
export-D56634321
branch
from
f2b1dda
to
4dcfd84
Compare
|
This pull request was exported from Phabricator. Differential Revision: D56634321 |
bernardbeckerman
pushed a commit
to bernardbeckerman/Ax
that referenced
this pull request
May 7, 2024
…ook#2432) Summary: Changes `observations_from_data` and `observations_from_map_data` signatures, replacing arg `fit_abandoned: bool` -> `statuses_to_fit` and `statuses_to_fit_map_metric` which are both `Optional[List[TrialStatus]]`. In `observations_from_data`, these default to `None` in which case `{TrialStatus.COMPLETED}` is used for any MapMetrics, else `NON_ABANDONED_STATUSES`, i.e., all trial statuses except abandoned. In `observations_from_map_data`, `NON_ABANDONED_STATUSES` are the default for all metrics. NOTE: As of this diff, any rows of `Data.df` containing `metric_names` that don't exist on `experiment` will be filtered out ([pointer](https://www.internalfb.com/diff/D56634321?permalink=330145676765780)). Reviewed By: saitcakmak Differential Revision: D56634321
bernardbeckerman
force-pushed
the
export-D56634321
branch
from
4dcfd84
to
18b0406
Compare
|
This pull request was exported from Phabricator. Differential Revision: D56634321 |
added 3 commits
May 7, 2024 06:18
…sponding experiments (facebook#2422) Summary: In D56634321, observations_from_dataframe fails if there are metric_names Data.df that don't also exist on the experiment. This adjusts tests so that they avoid this issue. Reviewed By: saitcakmak Differential Revision: D56850033
Summary: `Keys.PAIRWISE_PREFERENCE_QUERY` or `str(Keys.PAIRWISE_PREFERENCE_QUERY)` is commonly used when a string is expected, whereas I'm suspecting the intention is `Keys.PAIRWISE_PREFERENCE_QUERY.value`. This is causing issues in D56634321 which (for now) fails if any metrics in Data.df are not present on experiment when calling observations_from_dataframe. Reviewed By: Balandat Differential Revision: D56850035
…ook#2432) Summary: Changes `observations_from_data` and `observations_from_map_data` signatures, replacing arg `fit_abandoned: bool` -> `statuses_to_fit` and `statuses_to_fit_map_metric` which are both `Optional[List[TrialStatus]]`. In `observations_from_data`, these default to `None` in which case `{TrialStatus.COMPLETED}` is used for any MapMetrics, else `NON_ABANDONED_STATUSES`, i.e., all trial statuses except abandoned. In `observations_from_map_data`, `NON_ABANDONED_STATUSES` are the default for all metrics. NOTE: As of this diff, any rows of `Data.df` containing `metric_names` that don't exist on `experiment` will be filtered out ([pointer](https://www.internalfb.com/diff/D56634321?permalink=330145676765780)). Reviewed By: saitcakmak Differential Revision: D56634321
bernardbeckerman
force-pushed
the
export-D56634321
branch
from
18b0406
to
8353ed4
Compare
|
This pull request was exported from Phabricator. Differential Revision: D56634321 |
|
This pull request has been merged in 2d8fa34. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Changes
observations_from_dataandobservations_from_map_datasignatures, replacing argfit_abandoned: bool->statuses_to_fitandstatuses_to_fit_map_metricwhich are bothOptional[List[TrialStatus]]. Inobservations_from_data, these default toNonein which case{TrialStatus.COMPLETED}is used for any MapMetrics, elseNON_ABANDONED_STATUSES, i.e., all trial statuses except abandoned. Inobservations_from_map_data,NON_ABANDONED_STATUSESare the default for all metrics.NOTE: As of this diff, any rows of
Data.dfcontainingmetric_namesthat don't exist onexperimentwill be filtered out (pointer).Reviewed By: saitcakmak
Differential Revision: D56634321