SynergyNeRF

[ArXiv] [Project]

Official PyTorch implementation of SynergyNeRF, as presented in our paper:

Synergistic Integration of Coordinate Network and Tensorial Feature for Improving Neural Radiance Fields from Sparse Inputs (ICML2024)
Mingyu Kim¹, Jun-Seong Kim², Se-Young Yun^1† and Jin-Hwa Kim^3†
1KAIST AI, ²POSTECH EE, ³NAVER AI Lab.
(^† indicates corresponding authors)

Update

• Training code.
• Inference code.
• Datasets.

News

• (24.05.01) Our paper has been accepted in ICML 2024.
• (23.10.21) Our paper has been presented at NeurIPS2023 DeepInverse Workshop.

Environment Setup

# create conda environment
conda create --name SynergyNeRF python=3.9

# activate env
conda activate SynergyNeRF

# install pytorch >= 1.12 (e.g cu116)
pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116

# install packages
pip install -r requirements.txt

This implementation utilizes the code bases of TensoRF and HexPlane.

Dataset

For static NeRFs, this implementation utilizes NeRF Synthetic and TankandTemples.

# NeRF Synthetic: download and extract nerf_synthetic.zip
cd SynergyNeRF_3D
mkdir data
cd data
unzip nerf_synthetic.zip 

# tankandtemples: download and extract TankAndTemple.zip
unzip TankAndTemple.zip 

For dynamic NeRFs, this implementation utilizes D-NeRF Dataset.

# D-NeRF dataset: download and extract data.zip
cd SynergyNeRF_4D
mkdir data
cd data
unzip data.zip 

Quick Start

We provide training scripts based on config files.
First, we illustrate training static NeRFs for nerf_synthetic and tankandtemples.
To run this code on GPUs with 24GB of VRAM, you should use the configuration files located in config/SynergyNeRF/revised_cfg/.
If you want to run the code as described in the original paper,
please use the configuration files found in config/SynergyNeRF_disentangled/official/.

# training nerf_synthetic
cd SynergyNeRF_3D 
# scenes : {chair, drums, ficus, lego, hotdog, materials, mic, ship}
python main.py --config=config/SynergyNeRF/revised_cfg/8_views/{scene}.yaml

# training tankandtemples
# scenes : {barn, caterpillar, family, truck}
python main.py --config=config/SynergyNeRF/tankandtemples_cfgs/{scene}.yaml

Second, we illustrate training dynamic NeRFs for the D-NeRF dataset.

# training nerf_synthetic
cd SynergyNeRF_4D 
# scenes : {bouncingballs, hellwarrior, hook, jumpingjacks, lego, mutant, standup, trex}
python main.py --config=config/SynergyNeRF/official/{scene}.yaml

Bibliography

@InProceedings{kim2024synergistic,
  author    = {Kim, Mingyu and Kim, Jun Seong and Yun, Se Young and Kim, Jin Hwa},  
  title     = {Synergistic Integration of Coordinate Network and Tensorial Feature for Improving NeRFs from Sparse Inputs},  
  booktitle = {Proceedings of the 41th International Conference on Machine Learning},
  year      = {2024},
  series    = {Proceedings of Machine Learning Research},
  publisher = {PMLR},  
}

Acknowledgement

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) [No.2022-0-00641, XVoice: Multi-Modal Voice Meta Learning]. A portion of this work was carried out during an internship at NAVER AI Lab. We also extend our gratitude to ACTNOVA for providing the computational resources required.