metrics[4 and 5 refactored] save results and compare models [GSoC23 cont'd]

[jpcbertoldo]

Description

This is a continuation of Anomaly Segmentation Metrics for Anomalib — GSoC 2023 @ OpenVINO.

Updated PRs at https://gist.github.com/jpcbertoldo/12553b7eaa97cfbf3e55bfd7d1cafe88?permalink_comment_id=4681988#gistcomment-4681988

---

Replaces #9 and #10 after a refactor on the PImOResult, which became a dataclass incorporating the save and load methods.

Refactors

Return types

PImOResult becomes a dataclass holding all the information for pimo curves, and the aucs from AUPImO becomes an analogous AUPImOResult.

Others

• move demo notebooks to a common folder
• change some docstrings
• update the 1st notebook's text
• some validations
• From the previous PR:

Known issue: the plot functions in AULogPImO were not entirely encapsulated in a function. This was corrected in the next PR.

Adress this and move the logic of the plots from AULogPImO to a functional interface (classes practically just pass the arguments).

Save & Load

Make it possible to save and load pimo curves and their aucs.

PImOResult will save the curves' metadat (shared fpr type and bounds) all the returns (fprs, tprs, shared fpr, etc) in a dict of tensors in a .pt so the curves are fully recoverable.

AUPImOResult will save the auc's metadata (shared fpr type, bounds, thresholds of the bounds) in a JSON file so it can be human readable.

Compare models

Add utilities to compare two or more models on the same dataset with a per-image metric.

Two types of comparison are proposed: parametric (the metric values) and non-parametric (ranks, i.e. sorting of models). For each method, there is a statistical test (respec., paired t-test and wilcoxon signed rank test) and a plot that visually explains the test.

Plots can eventually show multiple (>2) methods, but the statistical tests are performed pairwise.
Results are read in a table with rows and columns are model names (model1 and model2) sorted by their average score or rank.

Statistical tests:

• Null hypothesis: methods are not significantly different.
• Alternative hypothesis: method1 (row) > method2 (column).
• Cell values are the "confidence to reject the null hypothesis" (1 - pvalue).

Example of parametric comparison (AULogPImO):

Example of non-parametric comparison from the parametric above:

---

List of refactors before merging feature branch:
https://github.com/jpcbertoldo/anomalib/blob/metrics/refactors/src/anomalib/utils/metrics/perimg/.refactors

Adds from this PR

-

---

Changes

• Bug fix (non-breaking change which fixes an issue)
• Refactor (non-breaking change which refactors the code base)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Checklist

• My code follows the pre-commit style and check guidelines of this project.
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing tests pass locally with my changes
• I have added a summary of my changes to the CHANGELOG (not for minor changes, docs and tests).

[jpcbertoldo]

Replaced by openvinotoolkit#1557 to catch up with v1.