Sync-NeRF : Generalizing Dynamic NeRFs
to Unsynchronized Videos

Official repository for "Sync-NeRF: Generalizing Dynamic NeRFs to Unsynchronized Videos"
enabling dynamic NeRFs to successfully reconstruct the scene from unsynchronized dataset.

Setup

We provide an integrated requirements file for Sync-MixVoxels and Sync-K-Planes.

pip install -r requirements.txt

You can download our Unsynchronized Dynamic Blender Dataset from this link

Sync-MixVoxels

As in the MixVoxels repository, please follow the below step:

Please note that in your first running, the above command will first pre-process the dataset, including resizing the frames by a factor of 2 (to 1K resolution which is a standard), as well as calculating the std of each video and save them into your disk. The pre-processing will cost about 2 hours, but is only required at the first running. After the pre-processing, the command will automatically train your scenes.

You can train the model using the following command:

python train.py --config path/to/config.txt

We also propose a method for optimizing time offsets during test time. You can execute this test-time optimization using the following command:

python train.py --config path/to/config.txt --test-optim

After completing model training, you can perform evaluation using the --render_only 1 flags.

python train.py --config path/to/config.txt --render_only 1 --ckpt path/to/checkpoint.pt

Sync-K-Planes

As in the K-Planes repository, please follow the below step:

Run first for 1 step with data_downsample=4 to generate weights for ray importance sampling

You can train the model using the following command:

PYTHONPATH='.' python plenoxels/main.py --config-path path/to/config.py

We also propose a method for optimizing time offsets during test time. You can execute this test-time optimization using the following command:

PYTHONPATH='.' python plenoxels/main.py --config-path path/to/config.py --log-dir path/to/logfolder --test_optim

After completing model training, you can perform evaluation using the --validate-only or --rendering-only flags.

Citation

@article{Kim2024Sync,
author = {Kim, Seoha and Bae, Jeongmin and Yun, Youngsik and Lee, Hahyun and Bang, Gun and Uh, Youngjung},
year = {2024},
month = {03},
pages = {2777-2785},
title = {Sync-NeRF: Generalizing Dynamic NeRFs to Unsynchronized Videos},
volume = {38},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
doi = {10.1609/aaai.v38i3.28057}
}

Acknowledge

The codes are based on MixVoxels and K-Planes, many thanks to the authors.