Hierarchical Average Precision Training for Pertinent Image Retrieval

This repo contains the official PyTorch implementation of the HAPPIER method as described in the ECCV 2022 paper: Hierarchical Average Precision Training for Pertinent Image Retrieval.

Suggested citation

Please consider citing our work:

@inproceedings{ramzi2022hierarchical,
  title={Hierarchical Average Precision Training for Pertinent Image Retrieval},
  author={Ramzi, Elias and Audebert, Nicolas and Thome, Nicolas and Rambour, Cl{\'e}ment and Bitot, Xavier},
  booktitle={European Conference on Computer Vision},
  pages={250--266},
  year={2022},
  organization={Springer}
}

Use HAPPIER

This will create a virtual environment and install the dependencies described in requirements.txt:

python3 -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install -e .

WARNING: as of now this code does not work for newer version of torch. It only works with torch==1.8.1.

Datasets

We use the following datasets for our paper:

• iNaturalist-2018 with splits
• Stanford Online Products
• DyML-datasets

Once extracted the code should work with the base structure of the datasets. You must precise the direction of the dataset to run an experiment:

dataset.data_dir=/Path/To/Your/Data/Stanford_Online_Products

For iNat you must put the split in the folder of the dataset: Inaturalist/Inat_dataset_splits.

You can also tweak the lib/expand_path.py function, as it is called for most path handling in the code.

Add you dataset

When implementing your custom dataset it shoud herit from BaseDataset

from happier.datasets.base_dataset import BaseDataset


class CustomDataset(BaseDataset):
  HIERARCHY_LEVEL = L

  def __init__(data_dir, mode, transform, **kwargs):
    self.paths = ...
    self.labels = ...  # should a numpy array of ndim == 2

    super().__init__(**kwargs)  # this should be at the end.

Then add you CustomDataset to the __init__.py file of datasets.

from .custom_dataset import CustomDataset

__all__ = [
    'CustomDataset',
]

Finally you should create a config file custom_dataset.yaml in happier/config/dataset.

Run the code

The code uses Hydra for the config. You can override arguments from command line or change a whole config. You can easily add other configs in happier/config.

Do not hesitate to create an issue if you have trouble understanding the configs, I will gladly answer you.

iNaturalist

iNat-base

CUDA_VISIBLE_DEVICES='0' python happier/run.py \
'experience.experiment_name=HAPPIER_iNat_base' \
'experience.log_dir=experiments/HAPPIER' \
experience.seed=0 \
experience.accuracy_calculator.compute_for_hierarchy_levels=[0,1] \
experience.warmup_step=5 \
optimizer=inat \
model=resnet_ln \
transform=inat \
dataset=inat_base \
loss=HAPPIER_inat

iNat-full

CUDA_VISIBLE_DEVICES='0' python happier/run.py \
'experience.experiment_name=HAPPIER_iNat_full' \
'experience.log_dir=experiments/HAPPIER/' \
experience.seed=0 \
experience.accuracy_calculator.compute_for_hierarchy_levels=[0,1,2,3,4,5,6] \
experience.warmup_step=5 \
optimizer=inat \
model=resnet_ln \
transform=inat \
dataset=inat_full \
loss=HAPPIER_inat

Stanford Online Products

SOP

CUDA_VISIBLE_DEVICES='0' python happier/run.py \
'experience.experiment_name=HAPPIER_SOP' \
'experience.log_dir=experiments/HAPPIER' \
experience.seed=0 \
experience.max_iter=100 \
experience.warmup_step=5 \
experience.accuracy_calculator.compute_for_hierarchy_levels=[0,1] \
optimizer=sop \
model=resnet_ln \
transform=sop \
dataset=sop \
loss=HAPPIER_SOP

Dynamic Metric Learning

DyML-Vehicle

CUDA_VISIBLE_DEVICES='0' python happier/run.py \
'experience.experiment_name=HAPPIER_dyml_vehicle' \
'experience.log_dir=experiments/HAPPIER' \
experience.seed=0 \
experience.accuracy_calculator.compute_for_hierarchy_levels=[0] \
experience.accuracy_calculator.overall_accuracy=True \
experience.accuracy_calculator.exclude=[NDCG,H-AP] \
experience.accuracy_calculator.recall_rate=[10,20] \
experience.accuracy_calculator.with_binary_asi=True \
optimizer=dyml \
model=dyml_resnet34 \
transform=dyml \
dataset=dyml_vehicle \
loss=HAPPIER

DyML-Animal

CUDA_VISIBLE_DEVICES='2' python happier/run.py \
'experience.experiment_name=HAPPIER_dyml_animal' \
'experience.log_dir=experiments/HAPPIER' \
experience.seed=0 \
experience.accuracy_calculator.compute_for_hierarchy_levels=[0] \
experience.accuracy_calculator.overall_accuracy=True \
experience.accuracy_calculator.exclude=[NDCG,H-AP] \
experience.accuracy_calculator.recall_rate=[10,20] \
experience.accuracy_calculator.with_binary_asi=True \
optimizer=dyml \
model=dyml_resnet34 \
transform=dyml \
dataset=dyml_animal \
loss=HAPPIER_5

DyML-Product

CUDA_VISIBLE_DEVICES='1' python happier/run.py \
'experience.experiment_name=HAPPIER_dyml_product' \
'experience.log_dir=experiments/HAPPIER' \
experience.seed=0 \
experience.max_iter=20 \
experience.warmup_step=5 \
experience.accuracy_calculator.compute_for_hierarchy_levels=[0,1,2] \
experience.accuracy_calculator.overall_accuracy=True \
experience.accuracy_calculator.exclude=[NDCG,H-AP] \
experience.accuracy_calculator.recall_rate=[10,20] \
experience.accuracy_calculator.with_binary_asi=True \
optimizer=dyml_product \
model=dyml_resnet34_product \
transform=dyml \
dataset=dyml_product \
loss=HAPPIER_product

Resources

Links to repo with useful features used for this code:

• Hydra: https://github.com/facebookresearch/hydra
• ROADMAP: https://github.com/elias-ramzi/ROADMAP
• NSM: https://github.com/azgo14/classification_metric_learning
• PyTorch: https://github.com/pytorch/pytorch
• Pytorch Metric Learning (PML): https://github.com/KevinMusgrave/pytorch-metric-learning

TODO LIST

• Add instruction to reproduce all experiments
• Make H-AP easier to use outside this repository
• Clean H-AP loss code
• Create paper with code badge