NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior

Project Page | Arxiv | Data | Pretrained Model

Wenjing Bian, Zirui Wang, Kejie Li, Jiawag Bian, Victor Adrian Prisacariu. (CVPR 2023 highlight)

Active Vision Lab, University of Oxford.

Table of Content

• Installation
• Data
• Usage
• Acknowledgement
• Citation

Installation

git clone https://github.com/ActiveVisionLab/nope-nerf.git
cd nope-nerf
conda env create -f environment.yaml
conda activate nope-nerf

Data and Preprocessing

1. Tanks and Temples: Our pre-processed Tanks and Temples data contains the 8 scenes shown in the paper. Each scene contains images, monocular depth estimations from DPT and COLMAP poses. You can download and unzip it to data directory.

2. NeRF LLFF: We also provide config file for NeRF LLFF dataset. You can download the dataset and unzip it to data directory. One example of the config file is configs/LLFF/fern.yaml.

3. If you want to use your own image sequence with customised camera intrinsics, you need to add an intrinsics.npz file to the scene directory. One example of the config file is configs/Test/images.yaml (please add your own data to the data/Test/images directory).

Monocular depth map generation: you can first download the pre-trained DPT model from this link provided by Vision Transformers for Dense Prediction to DPT directory, then run

python preprocess/dpt_depth.py configs/preprocess.yaml

to generate monocular depth maps. You need to modify the cfg['dataloading']['path'] and cfg['dataloading']['scene'] in configs/preprocess.yaml to your own image sequence.

Training

1. Train a new model from scratch:

python train.py configs/Tanks/Ignatius.yaml

where you can replace configs/Tanks/Ignatius.yaml with other config files.

You can monitor on http://localhost:6006 the training process using tensorboard:

tensorboard --logdir ./out --port 6006

For available training options, please take a look at configs/default.yaml.

Evaluation

1. Evaluate image quality and depth:

python evaluation/eval.py configs/Tanks/Ignatius.yaml

To evaluate depth: add --depth . Note that you need to add ground truth depth maps by yourself.

2. Evaluate poses:

python evaluation/eval_poses.py configs/Tanks/Ignatius.yaml

To visualise estimated & ground truth trajectories: add --vis

More Visualisations

Novel view synthesis

python vis/render.py configs/Tanks/Ignatius.yaml

Pose visualisation (estimated trajectory only)

python vis/vis_poses.py configs/Tanks/Ignatius.yaml

Acknowledgement

We thank Theo Costain and Michael Hobley for helpful comments and proofreading. We thank Shuai Chen and Xinghui Li for insightful discussions. Wenjing Bian is supported by China Scholarship Council (CSC).

We refer to NeRFmm, UNISURF, Vision Transformers for Dense Prediction, kitti-odom-eval and nerf-pytorch. We thank the excellent code they provide.

Citation

 @inproceedings{bian2022nopenerf,
	author    = {Wenjing Bian and Zirui Wang and Kejie Li and Jiawang Bian and Victor Adrian Prisacariu},
	title     = {NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior},
	journal   = {CVPR},
	year      = {2023}
	}