Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add F1 metric #6822

Merged
merged 3 commits into from
Jan 26, 2024
Merged

feat: Add F1 metric #6822

merged 3 commits into from
Jan 26, 2024

Conversation

awinml
Copy link
Contributor

@awinml awinml commented Jan 24, 2024

Related Issues

Fixes #6068

Proposed Changes:

Adds support for the F1 metric to EvaluationResult.calculate_metrics(...)

The _calculate_f1 method of EvaluationResult had been updated to compute the F1 metric:

def _calculate_f1(
    self,
    output_key: str,
    regexes_to_ignore=None,
    ignore_case=False,
    ignore_punctuation=False,
    ignore_numbers=False,
) -> MetricsResult:
     ....
     return MetricsResult({"f1": f1})

Usage:

For evaluation of a pipeline:

pipeline = Pipeline()
inputs = [...]
expected_outputs = [...]

eval_result = eval(pipeline, inputs=inputs, expected_outputs=expected_outputs)
f1_metric = eval_result.calculate_metrics(Metric.F1, output_key="answers")

How did you test it?

Unit tests have been added.
End-to-end tests with the following pipelines have been added:

  • Extractive QA Pipeline
  • RAG Pipeline with BM25 Retriever
  • RAG Pipeline with Embedding Retriever

This code was written collaboratively with @vrunm.

@awinml awinml requested review from a team as code owners January 24, 2024 12:00
@awinml awinml requested review from dfokina and silvanocerza and removed request for a team January 24, 2024 12:00
@github-actions github-actions bot added topic:tests 2.x Related to Haystack v2.0 type:documentation Improvements on the docs labels Jan 24, 2024
@coveralls
Copy link
Collaborator

coveralls commented Jan 24, 2024

Pull Request Test Coverage Report for Build 7640805943

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.03%) to 88.906%

Totals Coverage Status
Change from base Build 7629685083: 0.03%
Covered Lines: 4544
Relevant Lines: 5111

💛 - Coveralls

Copy link
Contributor

@silvanocerza silvanocerza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! 💯

Thanks for another great contribution. 😁

@silvanocerza silvanocerza merged commit 7217f9d into deepset-ai:main Jan 26, 2024
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.x Related to Haystack v2.0 topic:tests type:documentation Improvements on the docs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement function to calculate F1 metric
3 participants