Official repository for "Sync-NeRF: Generalizing Dynamic NeRFs to Unsynchronized Videos"
enabling dynamic NeRFs to successfully reconstruct the scene from unsynchronized dataset.
We provide an integrated requirements file for Sync-MixVoxels and Sync-K-Planes.
pip install -r requirements.txt
You can download our Unsynchronized Dynamic Blender Dataset from this link
As in the MixVoxels repository, please follow the below step:
Please note that in your first running, the above command will first pre-process the dataset, including resizing the frames by a factor of 2 (to 1K resolution which is a standard), as well as calculating the std of each video and save them into your disk. The pre-processing will cost about 2 hours, but is only required at the first running. After the pre-processing, the command will automatically train your scenes.
You can train the model using the following command:
python train.py --config path/to/config.txt
We also propose a method for optimizing time offsets during test time. You can execute this test-time optimization using the following command:
python train.py --config path/to/config.txt --test-optim
After completing model training, you can perform evaluation using the --render_only 1 flags.
python train.py --config path/to/config.txt --render_only 1 --ckpt path/to/checkpoint.pt
As in the K-Planes repository, please follow the below step:
Run first for 1 step with data_downsample=4 to generate weights for ray importance sampling
You can train the model using the following command:
PYTHONPATH='.' python plenoxels/main.py --config-path path/to/config.py
We also propose a method for optimizing time offsets during test time. You can execute this test-time optimization using the following command:
PYTHONPATH='.' python plenoxels/main.py --config-path path/to/config.py --log-dir path/to/logfolder --test_optim
After completing model training, you can perform evaluation using the --validate-only or --rendering-only flags.
@article{Kim2024Sync,
author = {Kim, Seoha and Bae, Jeongmin and Yun, Youngsik and Lee, Hahyun and Bang, Gun and Uh, Youngjung},
year = {2024},
month = {03},
pages = {2777-2785},
title = {Sync-NeRF: Generalizing Dynamic NeRFs to Unsynchronized Videos},
volume = {38},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
doi = {10.1609/aaai.v38i3.28057}
}
The codes are based on MixVoxels and K-Planes, many thanks to the authors.