PIMO #1726
PIMO #1726
Conversation
jpcbertoldo
requested review from
samet-akcay,
ashwinvaidya17 and
djdameln
as code owners
github-actions
bot
added
Dependencies
|
another unresolved issue from the previous PR [about "ATTENTION..." in docstrings]
how can i check that? |
The documentation is built here based on your changes |
there are some [metadata] stuff showing up, idk why apparently sphinx doesnt like dataclasses? |
Tree structure is also messed up a bit. It might be an idea to split each metric into a separate section. |
https://anomalib--1726.org.readthedocs.build/en/1726/markdown/guides/reference/metrics/index.html it's not quite working as expected
|
samet-akcay
added
Feature
and removed
Dependencies
There was a problem hiding this comment.
A lot has changed since this PR was submitted, but I've finally gotten around to reviewing it. This is a huge PR with a lot of efforts behind it. However, I have some concerns. I've gone over it once but I think I'll require a few more passes for a more thorough review. But meanwhile we can start the discussions for the current opens.
Signed-off-by: jpcbertoldo <[email protected]>
Signed-off-by: jpcbertoldo <[email protected]>
I removed the author tag and add the original code's repo to I think the decision for For the record (if i get it well), it only has two requirements: install_requires = [
'llvmlite >={},<{}'.format(min_llvmlite_version, max_llvmlite_version),
'numpy >={},<{}'.format(min_numpy_run_version, max_numpy_run_version),
'importlib_metadata; python_version < "3.9"',
]https://github.com/numba/numba/blob/d4460feb8c91213e7b89f97b632d19e34a776cd3/setup.py#L25-L28 I copied these from |
There was a problem hiding this comment.
I'm getting the following error when I try to use the AUPIMO metric to evaluate an anomalib model. Can you check it out?
ValueError: The `.compute()` return of the metric logged as 'pixel_AUPIMO' must be a tensor. Found (PIMOResult(shared_fpr_metric='mean-per-image-fpr'), AUPIMOResult(shared_fpr_metric='mean-per-image-fpr', fpr_lower_bound=1e-05, fpr_upper_bound=0.0001, num_threshs=48480))
| # Copyright (C) 2024 Intel Corporation | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| import numba |
There was a problem hiding this comment.
I'm also a bit hesitant to add the numba requirement. I can see the benefit that it brings, but at the same time it adds an unnecessary dependency and it increases the complexity of the code. Without Numba we could have a pure pytorch implementation of the metric, which would be much cleaner and more in line with the rest of the library.
This is normal because aupimo returns many values so it was encapsulated in that dataclass. |
I think that would be a good idea. With the default setting the metric should be fully compatible with Anomalib's pipeline. So users should be able to enable it from the config/cli or API to have Anomalib report the average AUPIMO value. |
Signed-off-by: jpcbertoldo <[email protected]>
Signed-off-by: jpcbertoldo <[email protected]>
Signed-off-by: jpcbertoldo <[email protected]>
@djdameln Done : ) i think the only missing issue is about numba (here #1726 (comment)) i can remove it if that's better, but it is already optional like you suggested in your last comment |
Signed-off-by: jpcbertoldo <[email protected]>
|
@samet-akcay could you launch a code check here please? About CDO, I think it is ok to apply the correction? But I'm not 100% sure (don't want to mess up the commit history 😬 ) |
done |
Signed-off-by: jpcbertoldo <[email protected]>
There was a problem hiding this comment.
@jpcbertoldo, thanks for creating this huuuge and amazing PR! Also, thanks for your patience. Took me a while to go through this.
I've got some comments/questions.
Clarity vs Conciseness in Naming
- When reading the code, I would prefer clarity over conciseness. I find the abbreviations hard to follow, which overall slows me down reading the code. Overall, I would prefer full words over abbreviations. Something like;
classification_curve.pyorbinary_classification_curve.py
thresholdsinstead ofthreshs
Validation
- I'm wondering how important the validation stuff for this metric evaluation?
- Is it possible to carry out these checks multiple times? Or just in the correct place.
- Is it possible to store validators in more organized way. currently there is a
_validation.py, but there are quite a bit of validators scattered across the sub-package.
Structure of the sub-package
-
I feel we could organise the sub-package a bit more. For example, instead of placing multiple modules horizontally like
pimo.py,pimo_numpy.py, we could potentially create a pimo sub package that organises these, which would be easier to follow. Similarly, it might be an idea to create something like utils/... , even maybecurves/..? -
For example, the actual PIMO implementation starts in Line 700ish. As a reader, I would expect to see
AUPIMO(Metric)inaupimo.pyandPIMO(Metric)inpimo.pyalmost right after imports. The rest of the stuff is sort of util to me
I'm working on something similar in PR #2305 to structure sub packages in data, for easier navigation
data
├── __init__.py
├── dataclasses
│ ├── __init__.py
│ ├── generic.py
│ ├── numpy
│ │ ├── __init__.py
│ │ ├── base.py
│ │ ├── depth.py
│ │ ├── image.py
│ │ └── video.py
│ └── torch
│ ├── __init__.py
│ ├── base.py
│ ├── depth.py
│ ├── image.py
│ └── video.py
├── datamodules
│ ├── __init__.py
│ ├── base
│ │ ├── __init__.py
│ │ ├── image.py
│ │ └── video.py
│ ├── depth
│ │ ├── __init__.py
│ │ ├── folder_3d.py
│ │ └── mvtec_3d.py
│ ├── image
│ │ ├── __init__.py
│ │ ├── btech.py
│ │ ├── folder.py
│ │ ├── kolektor.py
│ │ ├── mvtec.py
│ │ └── visa.py
│ └── video
│ ├── __init__.py
│ ├── avenue.py
│ ├── shanghaitech.py
│ └── ucsd_ped.py
├── datasets
│ ├── __init__.py
│ ├── base
│ │ ├── __init__.py
│ │ ├── depth.py
│ │ ├── image.py
│ │ └── video.py
│ ├── depth
│ │ ├── __init__.py
│ │ ├── folder_3d.py
│ │ └── mvtec_3d.py
│ ├── image
│ │ ├── __init__.py
│ │ ├── btech.py
│ │ ├── folder.py
│ │ ├── kolektor.py
│ │ ├── mvtec.py
│ │ └── visa.py
│ └── video
│ ├── __init__.py
│ ├── avenue.py
│ ├── shanghaitech.py
│ └── ucsd_ped.py
├── ...
├── transforms
│ └── ...
├── utils
│ └── ...
└── validators
├── __init__.py
├── numpy
│ ├── __init__.py
│ ├── depth.py
│ ├── image.py
│ └── video.py
├── path.py
└── torch
├── __init__.py
├── depth.py
├── image.py
└── video.pyGiven your other commitments these days, one possibility would be to work on it together by merging it to a feature branch? Any thoughts? @jpcbertoldo, @ashwinvaidya17, @djdameln ?
| path: str | Path, | ||
| base_dir: str | Path | None = None, | ||
| should_exist: bool = True, | ||
| accepted_extensions: tuple[str, ...] | None = None, |
There was a problem hiding this comment.
I think extensions would be sufficient in this case
| accepted_extensions: tuple[str, ...] | None = None, | |
| extensions: tuple[str, ...] | None = None, |
| """Validate the path. | ||
|
|
||
| Args: | ||
| path (str | Path): Path to validate. | ||
| base_dir (str | Path): Base directory to restrict file access. | ||
| should_exist (bool): If True, do not raise an exception if the path does not exist. | ||
| accepted_extensions (tuple[str, ...] | None): Accepted extensions for the path. An exception is raised if the |
There was a problem hiding this comment.
| accepted_extensions (tuple[str, ...] | None): Accepted extensions for the path. An exception is raised if the | |
| extensions (tuple[str, ...] | None): Accepted extensions for the path. An exception is raised if the |
| @@ -213,6 +220,11 @@ def validate_path(path: str | Path, base_dir: str | Path | None = None, should_e | |||
| msg = f"Read or execute permissions denied for the path: {path}" | |||
| raise PermissionError(msg) | |||
|
|
|||
| # Check if the path has one of the accepted extensions | |||
| if accepted_extensions is not None and path.suffix not in accepted_extensions: | |||
There was a problem hiding this comment.
| if accepted_extensions is not None and path.suffix not in accepted_extensions: | |
| if extensions is not None and path.suffix not in extensions: |
| # Copyright (C) 2024 Intel Corporation | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| from .binclf_curve import per_image_binclf_curve, per_image_fpr, per_image_tpr |
There was a problem hiding this comment.
I have a question regarding .binclf: Is there any case where the classification is not binary? If binary only, can we assume that the abbreviation bin is redundant?
During reading the code, I find it a bit hard to follow these abbreviations
|
|
||
|
|
||
| class BinclfThreshsChoice(Enum): | ||
| """Sequence of thresholds to use.""" |
There was a problem hiding this comment.
I think expanding the docstring would be good to explain what each choice would do. For example, why would a user, choose given over minmax_linspace, or mean_fpr_optimized
| return utils_numpy.compare_models_pairwise_ttest_rel(scores_per_model_with_arrays, alternative, higher_is_better) | ||
|
|
||
|
|
||
| def compare_models_pairwise_wilcoxon( |
| HI: str = "hi" | ||
| LO: str = "lo" |
There was a problem hiding this comment.
why not
HIGH, instead of HI, and LOW, instead LO?
| - 'image_idx': Index of the image in `per_image_scores` whose score is the closest to the statistic's value. | ||
| - 'score': The score of the image at index `image_idx` (not necessarily the same as `stat_value`). | ||
|
|
||
| The list is sorted by increasing `stat_value`. |
There was a problem hiding this comment.
As a reader/user, I would love to see an example to understand how to use this
| # =========================================== ARGS VALIDATION =========================================== | ||
|
|
||
|
|
||
| def _validate_is_anomaly_maps(anomaly_maps: Tensor) -> None: |
There was a problem hiding this comment.
would it be an idea to move these validation utils to _validate.py? these functions already use functions from validate.
|
|
||
| Returns: | ||
| PIMOResult: PIMO curves dataclass object. See `PIMOResult` for details. | ||
| """ |
There was a problem hiding this comment.
I think examples section here would be really useful. Like what do I need to run AUPIMO with some basic torch input and output
There was a problem hiding this comment.
I agree with Samet’s suggestion to move all validation checks to a separate module, as it would significantly improve the clarity and organization of pimo.py. Furthermore, I propose we also consider removing the NumPy-based computations. Since we aren’t using NumPy for other metrics in Anomalib, and there doesn’t seem to be a current need for NumPy in our metric computations, eliminating it could further simplify the codebase.
As Samet suggested, we can target this change to a feature branch so that the rest of the team can assist in reducing your workload, especially given the significant amount of work you’ve already done.
Ok, no worries. So shoul I remove the Just one warning. PIMO is fast to compute thanks to which is in numpy, so that specific one would be worth keeping. Using torchmetrics (even on GPU) is slower. The constraint is that the thresholds are shared across images, but the binary classification (for multiple thresholds) are computed per image. If this one is removed, then we need a torch-based implementation, which I'm sure is significantly slower for evaluation on the original resolution of images (1000-ish), which was a major argument in the paper. This is a topic we went through last year, but I can try to find the graphs to put numbers in this difference of execution time. So, do we keep or remove this one? ( |
|
Okay, then let's keep |
(just guessing) In the torchmetrics class, there isn't much happening in the |
ashwinvaidya17
merged commit 29cb912
into
openvinotoolkit:feature/pimo
📝 Description
Replace #1557, which replaces the PRs from https://gist.github.com/jpcbertoldo/12553b7eaa97cfbf3e55bfd7d1cafe88 .
Implements refactors from https://github.com/jpcbertoldo/anomalib/blob/metrics/refactors/src/anomalib/utils/metrics/perimg/.refactors .
arxiv: https://arxiv.org/abs/2401.01984
medium post: https://medium.com/p/c653ac30e802
GSoC deliverable: https://gist.github.com/jpcbertoldo/12553b7eaa97cfbf3e55bfd7d1cafe88
Closes #1728 1728
✨ Changes
Select what type of change your PR is:
✅ Checklist
Before you submit your pull request, please make sure you have completed the following steps:
For more information about code review checklists, see the Code Review Checklist.