This is an implementation of NeRF for MegEngine.
NeRF (Neural Radiance Fields) is a method that achieves state-of-the-art results for synthesizing novel views of complex scenes. Here are some videos generated by this repository (pre-trained models are provided below):
This project is a faithful MegEngine implementation of NeRF. The code is based on authors' Tensorflow implementation here and Pytorch implemention here by Yen-Chen Lin.
If you meet any problems when using this repo, please feel free to contact FateScript @ Megvii.
It's recommended to use venv to train your NeRF model.
A simple venv command example:
python3 -m venv nerf_mge
source nerf_mge/bin/activategit clone https://github.com/MegEngine/NeRF.git
cd nerf
python3 -m pip -v -e .
The LLFF data loader requires ImageMagick.
You will also need the LLFF code (and COLMAP) set up to compute poses if you want to run on your own real data.
Download data for two example datasets: lego and fern
bash download_example_data.sh
To train a low-res lego NeRF:
python3 run.py --config configs/lego.txt
After training for 100k iterations (~8 hours on a single 2080 Ti), you can find the following video at logs/lego_test/lego_test_spiral_100000_rgb.mp4.
To train a low-res fern NeRF:
python run.py --config configs/fern.txt
After training for 200k iterations (~8 hours on a single 2080 Ti), you can find the following video at logs/fern_test/fern_test_spiral_200000_rgb.mp4 and logs/fern_test/fern_test_spiral_200000_disp.mp4
To play with other scenes presented in the paper, download the data here. Place the downloaded dataset according to the following directory structure:
├── configs
│ ├── ...
│
├── data
│ ├── nerf_llff_data
│ │ └── fern
│ │ └── flower # downloaded llff dataset
│ │ └── horns # downloaded llff dataset
| | └── ...
| ├── nerf_synthetic
| | └── lego
| | └── ship # downloaded synthetic dataset
| | └── ...
To train NeRF on different datasets:
python run.py --config configs/{DATASET}.txt
replace {DATASET} with trex | horns | flower | fortress | lego | etc.
To test NeRF trained on different datasets:
python run.py --config configs/{DATASET}.txt --render_only
replace {DATASET} with trex | horns | flower | fortress | lego | etc.
You can download the pre-trained models here. Place the downloaded directory in ./logs in order to test it later. See the following directory structure for an example:
├── logs
│ ├── fern_test
│ ├── flower_test # downloaded logs
│ ├── trex_test # downloaded logs
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Ben Mildenhall*1,
Pratul P. Srinivasan*1,
Matthew Tancik*1,
Jonathan T. Barron2,
Ravi Ramamoorthi3,
Ren Ng1
1UC Berkeley, 2Google Research, 3UC San Diego
*denotes equal contribution
A neural radiance field is a simple fully connected network (weights are ~5MB) trained to reproduce input views of a single scene using a rendering loss. The network directly maps from spatial location and viewing direction (5D input) to color and opacity (4D output), acting as the "volume" so we can use volume rendering to differentiably render new views
Kudos to the authors for their amazing results:
@misc{mildenhall2020nerf,
title={NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis},
author={Ben Mildenhall and Pratul P. Srinivasan and Matthew Tancik and Jonathan T. Barron and Ravi Ramamoorthi and Ren Ng},
year={2020},
eprint={2003.08934},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
Some code in this repo is based on the following repo:
- https://github.com/bmild/nerf MIT License
- https://github.com/yenchenlin/nerf-pytorch MIT License
- Some operators is not supported well in MegEngine, I work around this issue by using operator writing in Python, this might cause NeRF-megengine runs slower.