NeRFMatch

This repository contains the code release of our paper accepted at ECCV2024:

The NeRFect Match: Exploring NeRF Features for Visual Localization. [Project Page | Paper | Poster]

Installation

Clone this repository and create a conda envoirnment with the following commands:

# Create conda env
conda env create -f configs/conda/nerfmatch_env.yml
conda activate neumatch
pip install -r configs/conda/requirements.txt

# Install this repo
pip install -e .

Data Preparation

• Download the 7-Scenes dataset from this link and place them under data/7scenes .

• Download the Cambridge Landmarks scenes (Great Court, Kings College, Old Hospital, Shop Facade, St. Marys Church) and place them under data/cambrdige.

• Execute the following command to download our pre-process data annotations and image retrieval pairs and SAM masks on Cambridge Landmarks for NeRF training. The Cambridge Landmarks annotations are converted from original dataset nvm file. 7-Scenes sfm ground truth json files are converted from pgt/sfm/7scenes.

cd data/
bash download_data.sh
cd  ..

• Execute the following command to download our pretrained nerf and nerfmatch models.

cd pretrained/
bash download_pretrained.sh
cd  ..

After those preparation steps, your data/ directory shall look like:

data
├── 7scenes
│   ├── chess
│   └── ...
├── annotations
│   └── 7scenes_jsons/sfm
│       ├── transforms_*_test.json
│       ├── transforms_*_train.json
│       └── ...
├── cambridge
│   ├── GreatCourt
│   └── ...
├── mask_preprocessed
│   └── cambridge
└── pairs
    ├── 7scenes    
    └── cambridge

Training and Evaluation

We refer users to model_train/README.md and model_eval/README.md for training and evaluation instructions.

Licenses

The source code is released under NVIDIA Source Code License v1. The pretrained models are released under CC BY-NC-SA 4.0.

Citation

If you are using our method, please cite:

@article{zhou2024nerfmatch,
  title={The NeRFect match: Exploring NeRF features for visual localization},
  author={Zhou, Qunjie and Maximov, Maxim and Litany, Or and Leal-Taix{\'e}, Laura},
  journal={European Conference on Computer Vision},
  year={2024}
}