feat: Add calculate_metrics and MetricsResult

[awinml]

Related Issues

• fixes Implement MetricsResult class #6062
• fixes Implement function to calculate metrics from EvaluationResult #6063

Proposed Changes:

Based on the design in the Evaluation proposal (#5794), we have implemented the following classes:

calculate_metrics:

def calculate_metrics(self, metric: Union[Metric, Callable[..., MetricsResult]], **kwargs) -> MetricsResult:
        ...
        return metric(self, **kwargs)

The calculate_metrics method of EvaluationResult computes the evaluation metric based on provided metric.
The method returns a MetricsResult object.

MetricsResult:

class MetricsResult(dict):

MetricsResult stores the metric values computed during the evaluation. It inherits from dict.
A save method has been implemented to save the metrics to a JSON file.

Metric:

class Metric(Enum):

Metric contains a list of standard metrics available. It inherits from enum.

How did you test it?

The following pipelines were used to test the evaluation using a placeholder metric value:

• Extractive QA Pipeline
• RAG Pipeline with BM25 Retriever
• RAG Pipeline with Embedding Retriever

The other tests will be added with the corresponding metric implementations.

This code was written collaboratively with @vrunm.

[silvanocerza]

Hey @awinml can you maybe split this into multiple PRs? Ideally one for each issue should be enough, it will speed up review time quite a bit. :)

[awinml]

@silvanocerza Thanks! I have moved the Exact Match implementation to a separate PR (#6696).

I have decided to keep both calculate_metrics and MetricsResult in this PR, as compared to splitting them, since they are fairly straightforward classes and mostly have placeholders for the metrics at the moment.

[silvanocerza]

Updated branch to fix tests failures.

[coveralls]

Pull Request Test Coverage Report for Build 7472477307

Warning: This coverage report may be inaccurate.

We've detected an issue with your CI configuration that might affect the accuracy of this pull request's coverage report.
To ensure accuracy in future PRs, please see these guidelines.
A quick fix for this PR: rebase it; your next report should be accurate.

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-0.3%) to 87.066%

---

Totals
Change from base Build 7458712685:	-0.3%
Covered Lines:	4214
Relevant Lines:	4840

---

💛 - Coveralls

[silvanocerza]

Sorry, I merged base in again before noticing you already did. 🤦

[silvanocerza]

I simplified a bit the logic to run supported metrics and added some tests so coverage doesn't go down. This is good to merge as soon as tests are green. 👍