Official PyTorch implementation of SynergyNeRF, as presented in our paper:
Synergistic Integration of Coordinate Network and Tensorial Feature for Improving Neural Radiance Fields from Sparse Inputs (ICML2024)
Mingyu Kim1, Jun-Seong Kim2,
Se-Young Yun1†
and Jin-Hwa Kim3†
1KAIST AI, 2POSTECH EE, 3NAVER AI Lab.
(† indicates corresponding authors)
- Training code.
- Inference code.
- Datasets.
- (24.05.01) Our paper has been accepted in ICML 2024.
- (23.10.21) Our paper has been presented at NeurIPS2023 DeepInverse Workshop.
# create conda environment
conda create --name SynergyNeRF python=3.9
# activate env
conda activate SynergyNeRF
# install pytorch >= 1.12 (e.g cu116)
pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116
# install packages
pip install -r requirements.txt
This implementation utilizes the code bases of TensoRF and HexPlane.
For static NeRFs, this implementation utilizes NeRF Synthetic and TankandTemples.
# NeRF Synthetic: download and extract nerf_synthetic.zip
cd SynergyNeRF_3D
mkdir data
cd data
unzip nerf_synthetic.zip
# tankandtemples: download and extract TankAndTemple.zip
unzip TankAndTemple.zip
For dynamic NeRFs, this implementation utilizes D-NeRF Dataset.
# D-NeRF dataset: download and extract data.zip
cd SynergyNeRF_4D
mkdir data
cd data
unzip data.zip
We provide training scripts based on config files.
First, we illustrate training static NeRFs for nerf_synthetic and tankandtemples.
To run this code on GPUs with 24GB of VRAM, you should use the configuration files located in config/SynergyNeRF/revised_cfg/
.
If you want to run the code as described in the original paper,
please use the configuration files found in config/SynergyNeRF_disentangled/official/
.
# training nerf_synthetic
cd SynergyNeRF_3D
# scenes : {chair, drums, ficus, lego, hotdog, materials, mic, ship}
python main.py --config=config/SynergyNeRF/revised_cfg/8_views/{scene}.yaml
# training tankandtemples
# scenes : {barn, caterpillar, family, truck}
python main.py --config=config/SynergyNeRF/tankandtemples_cfgs/{scene}.yaml
Second, we illustrate training dynamic NeRFs for the D-NeRF dataset.
# training nerf_synthetic
cd SynergyNeRF_4D
# scenes : {bouncingballs, hellwarrior, hook, jumpingjacks, lego, mutant, standup, trex}
python main.py --config=config/SynergyNeRF/official/{scene}.yaml
@InProceedings{kim2024synergistic,
author = {Kim, Mingyu and Kim, Jun Seong and Yun, Se Young and Kim, Jin Hwa},
title = {Synergistic Integration of Coordinate Network and Tensorial Feature for Improving NeRFs from Sparse Inputs},
booktitle = {Proceedings of the 41th International Conference on Machine Learning},
year = {2024},
series = {Proceedings of Machine Learning Research},
publisher = {PMLR},
}
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) [No.2022-0-00641, XVoice: Multi-Modal Voice Meta Learning]. A portion of this work was carried out during an internship at NAVER AI Lab. We also extend our gratitude to ACTNOVA for providing the computational resources required.