-
Notifications
You must be signed in to change notification settings - Fork 289
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ADD] Optimise on custom metric #486
base: development
Are you sure you want to change the base?
Conversation
Codecov ReportBase: 85.28% // Head: 28.03% // Decreases project coverage by
Additional details and impacted files@@ Coverage Diff @@
## development #486 +/- ##
================================================
- Coverage 85.28% 28.03% -57.26%
================================================
Files 232 232
Lines 16456 16467 +11
Branches 3048 2733 -315
================================================
- Hits 14034 4616 -9418
- Misses 1573 11849 +10276
+ Partials 849 2 -847
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
############################################################################### | ||
# Define and add custom score function | ||
# ==================================== | ||
def score_function(y_test, y_pred): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For Time Series Tasks, an additional kwargs must be attached:
https://github.com/sktime/sktime/blob/main/sktime/performance_metrics/forecasting/_functions.py#L179
Should we create a new example for this, or do we add another docstring here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll add it to the same example. We can divide this example into two parts- for tabular tasks and for time series tasks
This PR enables the user to add their own custom metric.
Types of changes
Note that a Pull Request should only contain one of refactoring, new features or documentation changes.
Please separate these changes and send us individual PRs for each.
For more information on how to create a good pull request, please refer to The anatomy of a perfect pull request.
Checklist:
Description
This PR adds a function,
add_metric
which allows the user to pass any custom metric and make it available to AutoPyTorch for HPO. It also includes an example on how to use this functionality.Motivation and Context
Fixes #477
How has this been tested?
I have added a test which checks if the given metric is available in the metrics for the current task type.