Skip to content

donydchen/matchnerf

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

sculpture printer





MatchNeRF

Official PyTorch implementation for MatchNeRF, a new generalizable NeRF approach that employs explicit correspondence matching as the geometry prior and can perform novel view synthesis on unseen scenarios with as few as two source views as input, without requiring any retraining and fine-tuning.

Explicit Correspondence Matching for Generalizable Neural Radiance Fields
Yuedong Chen1, Haofei Xu2, Qianyi Wu1, Chuanxia Zheng3, Tat-Jen Cham4, Jianfei Cai1
1Monash University, 2ETH Zurich, 3University of Oxford, 4Nanyang Technological University
arXiv 2023

Recent Updates
  • 25-Apr-2023: released MatchNeRF codes and models.


Table of Contents

Setup Environment

This project is developed and tested on a CUDA11 device. For other CUDA version, manually update the requirements.txt file to match the settings before preceding.

git clone --recursive https://github.com/donydchen/matchnerf.git
cd matchnerf
conda create --name matchnerf python=3.8
conda activate matchnerf
pip install -r requirements.txt

For rendering video output, it requires ffmpeg to be installed on the system, you can double check by running ffmpeg -version. If ffmpeg does not exist, consider installing it by running conda install ffmpeg.

Download Datasets

DTU (for both training and testing)

  • Download the preprocessed DTU training data dtu_training.rar and Depth_raw.zip from original MVSNet repo.

  • Extract 'Cameras/' and 'Rectified/' from the above downloaded 'dtu_training.rar', and extract 'Depths' from the 'Depth_raw.zip'. Link all three folders to data/DTU, which should then have the following structure

data/DTU/
    |__ Cameras/
    |__ Depths/
    |__ Rectified/

Blender (for testing only)

Real Forward Facing (for testing only)

Testing

MVSNeRF Setting (3 Nearest Views)

Download the pretrained model matchnerf_3v.pth and save to configs/pretrained_models/matchnerf_3v.pth, then run

python test.py --yaml=test --name=matchnerf_3v

If encounters CUDA out-of-memory, please reduce the ray sampling number, e.g., append --nerf.rand_rays_test==4096 to the command.

Performance should be exactly the same as below,

Dataset PSNR SSIM LPIPS
DTU 26.91 0.934 0.159
Real Forward Facing 22.43 0.805 0.244
Blender 23.20 0.897 0.164

Training

Download the GMFlow pretrained weight (gmflow_sintel-0c07dcb3.pth) from the original GMFlow repo, and save it to configs/pretrained_models/gmflow_sintel-0c07dcb3.pth, then run

python train.py --yaml=train

Rendering Video

python test.py --yaml=test_video --name=matchnerf_3v_video

Results (without any per-scene fine-tuning) should be similar as below,

Visual Results

dtu_scan38_view24
DTU: scan38_view24

blender_materials_view36
Blender: materials_view36

llff_leaves_view13
Real Forward Facing: leaves_view13

Use Your Own Data

  • Download the model (matchnerf_3v_ibr.pth) pretrained with IBRNet data (follow 'GPNR Setting 1'), and save it to configs/pretrained_models/matchnerf_3v_ibr.pth.
  • Following the instructions detailed in the LLFF repo, use img2poses.py to recover camera poses.
  • Update the colmap data loader at datasets/colmap.py accordingly.

We provide the following 3 input views demo for your reference.

# lower resolution but fast
python test.py --yaml=demo_own
# full version
python test.py --yaml=test_video_own

The generated video will look like,

colmap_printer
Demo: own data, printer

Miscellaneous

Citation

If you use this project for your research, please cite our paper.

@article{chen2023matchnerf,
    title={Explicit Correspondence Matching for Generalizable Neural Radiance Fields},
    author={Chen, Yuedong and Xu, Haofei and Wu, Qianyi and Zheng, Chuanxia and Cham, Tat-Jen and Cai, Jianfei},
    journal={arXiv preprint arXiv:2304.12294},
    year={2023}
}

Pull Request

You are more than welcome to contribute to this project by sending a pull request.

Acknowledgments

This implementation borrowed many code snippets from GMFlow, MVSNeRF, BARF and GIRAFFE. Many thanks for all the above mentioned projects.

About

🖨️[arXiv'23] Official PyTorch Implementation of MatchNeRF

Topics

Resources

License

Stars

Watchers

Forks

Contributors 4

  •  
  •  
  •  
  •  

Languages