-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add calculate_metrics
and MetricsResult
#6680
feat: Add calculate_metrics
and MetricsResult
#6680
Conversation
Hey @awinml can you maybe split this into multiple PRs? Ideally one for each issue should be enough, it will speed up review time quite a bit. :) |
calculate_metrics
, MetricsResult
, Exact Match Metriccalculate_metrics
and MetricsResult
@silvanocerza Thanks! I have moved the Exact Match implementation to a separate PR (#6696). I have decided to keep both |
Updated branch to fix tests failures. |
Pull Request Test Coverage Report for Build 7472477307Warning: This coverage report may be inaccurate.We've detected an issue with your CI configuration that might affect the accuracy of this pull request's coverage report.
💛 - Coveralls |
Sorry, I merged base in again before noticing you already did. 🤦 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I simplified a bit the logic to run supported metrics and added some tests so coverage doesn't go down. This is good to merge as soon as tests are green. 👍
Related Issues
MetricsResult
class #6062EvaluationResult
#6063Proposed Changes:
Based on the design in the Evaluation proposal (#5794), we have implemented the following classes:
calculate_metrics:
The
calculate_metrics
method ofEvaluationResult
computes the evaluation metric based on provided metric.The method returns a
MetricsResult
object.MetricsResult:
MetricsResult
stores the metric values computed during the evaluation. It inherits fromdict
.A
save
method has been implemented to save the metrics to a JSON file.Metric:
Metric
contains a list of standard metrics available. It inherits fromenum
.How did you test it?
The following pipelines were used to test the evaluation using a placeholder metric value:
The other tests will be added with the corresponding metric implementations.
This code was written collaboratively with @vrunm.