This repository contains the implementation of the paper: Online Invariance Selection for Local Feature Descriptors, R. Pautrat, V. Larsson, M. Oswald and M. Pollefeys (Oral at ECCV 2020).
LISRD offers the possibility to leverage descriptors with different invariances (e.g. invariant or variant to rotation and/or illumination) and to perform an online selection of the most adapted invariance when matching two images.
The following shows for example matches predicted with LISRD-SIFT that selects descriptors between SIFT (rotation invariant) and Upright SIFT (rotation variant):
Clone the repository with its submodule:
git clone --recurse-submodules https://github.com/rpautrat/LISRD.gitWe recommend using this code in a Python environment (e.g. venv or conda). The following script installs the necessary requirements and install the repository as a Python package locally:
make installAll the training parameters should be present in a configuration file located in the folder lisrd/configs. The generic script to train your own model from the root folder is the following:
python -m lisrd.experiment train <path to your config file> <path to your experiment>For example, training LISRD with the 4 types of invariance used in the paper would be:
python -m lisrd.experiment train lisrd/configs/lisrd.yaml ~/Documents/experiments/My_experimentUse the config file lisrd/configs/lisrd_sift.yaml to instead train LISRD-SIFT that chooses between SIFT and Upright SIFT descriptors.
We provide two pretrained models:
- lisrd_aachen: version of LISRD used in the paper, trained on the MS COCO dataset, Multi-Illumination Images in the Wild and Aachen dataset images augmented with style-transfer or with optical flow information.
- lisrd_vidit: new version of LISRD, trained on the MS COCO dataset, Multi-Illumination Images in the Wild and the Virtual Image Dataset for Illumination Transfer (VIDIT). The results below are obtained using this model.
We provide a notebook showing how to use the trained models of LISRD. Additionally, lisrd/export_features.yaml is a script to export the LISRD descriptors with either SIFT or SuperPoint keypoints on a given set of images. It can be used as follows:
python -m lisrd.export_features <path to a txt file listing all your images> <name of the model (lisrd or lisrd_sift)> --checkpoint <path to checkpoint> --keypoints <type of keypoints (sift or superpoint)> --num_kp <number of keypoints (default: 2000)>The Rotated Day-Night Image Matching (RDNIM) dataset originates from the DNIM dataset and has been augmented with homographic warps with 50% of the images including rotations. The images used for evaluation in the paper are available here.
Comparison to the state of the art on the RDNIM dataset, using SuperPoint keypoints for all methods and a correctness threshold of 3 pixels:
| Day reference | Night reference | |||||
|---|---|---|---|---|---|---|
| Homography estimation | Precision | Recall | Homography estimation | Precision | Recall | |
| Root SIFT | 0.134 | 0.184 | 0.125 | 0.186 | 0.239 | 0.182 |
| HardNet | 0.249 | 0.225 | 0.224 | 0.325 | 0.359 | 0.365 |
| SOSNet | 0.226 | 0.218 | 0.226 | 0.252 | 0.288 | 0.296 |
| SuperPoint | 0.178 | 0.191 | 0.214 | 0.235 | 0.259 | 0.296 |
| D2-Net | 0.124 | 0.196 | 0.145 | 0.195 | 0.265 | 0.218 |
| R2D2 | 0.190 | 0.173 | 0.180 | 0.229 | 0.237 | 0.237 |
| GIFT | 0.225 | 0.155 | 0.149 | 0.294 | 0.240 | 0.229 |
| LISRD (ours) | 0.318 | 0.406 | 0.439 | 0.391 | 0.488 | 0.520 |
If you use this code in your project, please consider citing the following paper:
@InProceedings{Pautrat_2020_ECCV,
author = {Pautrat, Rémi and Larsson, Viktor and Oswald, Martin R. and Pollefeys, Marc},
title = {Online Invariance Selection for Local Feature Descriptors},
booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
year = {2020},
}