This is the official repository for the 3DV 2019 paper Sparse-To-Dense Hypercolumn Matching for Long-Term Visual Localization. We introduce propose a novel approach to feature point matching, suitable for robust and accurate outdoor visual localization in long-term scenarios. The proposed solution achieves state-of-the-art accuracy on several outdoor datasets, in challenging categories such as day-to-night or cross-seasonal changes.
The proposed approach was ranked 2nd in the visual localization challenge of the CVPR 2019 challenge on Long-Term Visual Localization using this codebase, with state-of-the-art results on nighttime and rural environments.
Inlier visualization for (from left to right) Superpoint sparse-to-sparse matching, Superpoint detection with hypercolumn sparse-to-sparse matching, and sparse-to-dense hypercolumn matching.
Run the following commands to install this repository and the required dependencies:
git clone https://github.com/germain-hug/S2DHM.git
cd S2D_Hypercolumn_Matching/
git submodule update --init --recursive
pip3 install -r requirements.txt
mkdir -p data/triangulation
This code was run and tested on Python 3.7.3, using Pytorch 1.0.1.post2 although it should be compatible with some previous versions. You can follow instructions to install Pytorch here.
To run this code, you will first need to download either RobotCar-Seasons or Extended CMU-Seasons from the website of the CVPR 2019 Challenge. Once unpacked, the root of the dataset should be updated accordingly in s2dm/configs/datasets/<your_dataset>.gin
.
In addition, we provide a pre-computed reconstruction of both datasets computed using SuperPoint. These triangulations were obtained using scripts borrowed from HF-Net, please to their repository for more details. The triangulation .npz
files can be downloaded from this link, and should be placed under data/triangulation/
.
The pre-trained weights for the main image retrieval network can be found under
checkpoints/
.
Overview of our sparse-to-dense hypercolumn matching pipeline.
You can run either of the following modes:
This mode predicts the query pose based as the pose of the top-ranked image in the database.
python3 run.py --dataset [robotcar|cmu] --mode nearest_neighbor
This mode predicts the query pose using sparse-to-sparse matching with SuperPoint detections and features in the query image.
python3 run.py --dataset [robotcar|cmu] --mode superpoint
This mode performs sparse-to-dense hypercolumn matching, as per the figure above.
python3 run.py --dataset [robotcar|cmu] --mode sparse_to_dense
After running, a .txt
file is produced and saved under results/
. This is the file that should be uploaded to the CVPR 2019 Visual Localization Challenge website to obtain the quantitative results.
To export image logs in logs/
(to allow for visual comparison of the approach) you can add the --log_images
argument when running the pipeline.
This codebase uses the gin configuration file system to store all high-level parameters.
All configuration files can be found under s2dm/configs/
and can be used to update most of the hyper-parameters and data paths.
At runtime, hyperparameters being used will be printed.
This repository provides code to run the pipeline on two dataset: RobotCar-Seasons and Extended CMU-Seasons. It can however be setup for your own dataset.
To do so, you should create a new class inheriting BaseDataset
and stored under s2dm/datasets/<your_dataset>.py
. You can look at the robotcar_dataset.py
or cmu_dataset.py
for inspiration.
Then, you will have to add a new config file for your dataset in s2dm/datasets/<your_dataset>.gin
and reference it in a new run gin config file under s2dm/runs/run_<mode>_on<your_dataset>.gin
.
Lastly, you will have to make sure that you have a reconstruction .npz
file stored in data/triangulation/
. Please refer to the HF-Net repository to compute such files using COLMAP.
Please consider citing the corresponding publication if you use this work:
@inproceedings{germain2019sparsetodense,
title={Sparse-To-Dense Hypercolumn Matching for Long-Term Visual Localization},
author={Germain, H. and Bourmaud, G. and Lepetit, V.},
article={International Conference on 3D Vision (3DV)},
year={2019}
}