Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement function to calculate metrics from EvaluationResult #6063

Closed
Tracked by #6061
silvanocerza opened this issue Oct 16, 2023 · 0 comments · Fixed by #6680
Closed
Tracked by #6061

Implement function to calculate metrics from EvaluationResult #6063

silvanocerza opened this issue Oct 16, 2023 · 0 comments · Fixed by #6680
Labels
2.x Related to Haystack v2.0 P2 Medium priority, add to the next sprint if no P1 available type:feature New feature or request

Comments

@silvanocerza
Copy link
Contributor

As specified in proposal #5794 we need to implement the calculate_metrics() function.

The calculated_metrics() function will expect this arguments:

  • evaluation_result: EvaluationResult
  • metric: Union[Metric, MetricCalculator]
  • **kwargs

And will return an instance of MetricsResult.

Metric can be either a simple Enum or a Literal. Probably an Enum will ease the discoverability of supported metrics.
MetricCalculator will be a Callable[..., MetricResult].
**kwargs will be passed to the function used to calculate the metrics, can be obviously omitted.

For more detailed information check out the original proposal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.x Related to Haystack v2.0 P2 Medium priority, add to the next sprint if no P1 available type:feature New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants