Xiaoyun Zheng1,2, Liwei Liao1,2, Xufeng Li3,
Jianbo Jiao4, Rongjie Wang2, Feng Gao1, Shiqi Wang3, Ronggang Wang1*
1 Peking University 2Peng Cheng Laboratory 3City University of Hong Kong 4University of Birmingham
*Corresponding author
PKU-DyMVHumans is a versatile human-centric dataset designed for high-fidelity reconstruction and rendering of dynamic human performances in markerless multi-view capture settings. It comprises 32 humans across 45 different dynamic scenarios, each featuring highly detailed appearances and complex human motions.
Inspired by recent advancements in neural radiance field (NeRF)-based scene representations, we carefully set up an off-the-shelf framework that is easy to provide those state-of-the-art NeRF-based implementations and benchmark on PKU-DyMVHumans dataset. This includes neural scene decomposition, 3D human reconstruction, and novel view synthesis of dynamic scenes.
The PKU-DyMVHumans dataset is available in Baidu Wangpan,【提取码g2xw】 and Hugging Face.
Part1: contains 8 scenarios that can be directly used for benchmarks in fine-grained foreground/background decomposition, 3D human reconstruction and novel view synthesis.
Part2: contains 37 scenarios with multi-view video sequences.
Note that by downloading the datasets, you acknowledge that you have read the agreement, understand it, and agree to be bound by them:
The PKU-DyMVHumans dataset is made available only for non-commercial research purposes. Any other use, in particular any use for commercial purposes, is prohibited.
You agree not to further copy, publish or distribute any portion of the dataset.
Peking University reserves the right to terminate your access to the dataset at any time.
For each scene, we provide the multi-view images (./case_name/per_view/cam_*/images/
), the coarse foreground with RGBA channels (./case_name/per_view/cam_*/images/
), as well as the coarse foreground segmentation (./case_name/per_view/cam_*/pha/
), which are obtained using BackgroundMattingV2.
To make the benchmarks easier compare with our dataset, we save different data formats (i.e., Surface-SOS, NeuS, NeuS2, Instant-ngp, and 3D-Gaussian) of PKU-DyMVHumans at Part1 and write a document that describes the data process.
.
|--- <case_name>
| |--- cams
| |--- videos
| |--- per_view
| |--- per_frame
| |--- data_ngp
| |--- data_NeuS
| |--- data_NeuS2
| |--- data_COLMAP
| |--- <overview_fme_*.png>
|--- ...
Optionally, camera parameters are provided in ./case_name/cams/
, the *_cam.txt
files are for example :
extrinsic
-0.9870368172239224 -0.022593485597630164 -0.15889573893915476 0.2776553077243575
0.023645162225597843 0.9587670058488046 -0.28320793562159163 0.6898850210976338
0.1587426462794277 -0.2832937749126657 -0.9458041072801162 4.304908547693294
0.0 0.0 0.0 1.0
intrinsic
1795.4695513117783 0.0 960.0
0.0 1908.5821699983292 540.0
0.0 0.0 1.0
<depth_value1> <depth_value2> <depth_value3> <depth_value4> # depth range and interval
To filter the scenarios to be downloaded depending on the purpose, you can take advantage of the format of the case names. The case name contains information regarding the duration, action, scenarios type, peeple per frame, and the gender. For instance,
We provide several scripts in this repo for you to experiment with our dataset. More detailed instructions are included in the files.
data_preprocess.sh
: Per view image overview, and save the image sequence and matting results.process2colmap2neus.sh
: process2colmap2neus.sh: Init data format for NeuS/NeuS2.process2nerf.sh
: Init data format for Instant-ngp/torch-ngp.
Also, we provide a converter script run_colmap.sh
, using the open source COLMAP software to extract SfM information and the necessary camera data.
Note that all require Python 3.7 or higher to be installed?and make sure that you have installed COLMAP and that it is available in your PATH. If you are using a video file as input, also be sure to install FFmpeg and make sure that it is available in your PATH.
Run the NeuS on PKU-DyMVHumans
NeuS supports the data format provided by data_NeuS.
.
<data_NeuS>
|---000000 # frame
| |---images
| | |---000.png
| | |---001.png
| | |---...
| |---mask
| | |---000.png
| | |---001.png
| | |---...
| |---cameras_sphere.npz
|---...
Run the NeuS2 on PKU-DyMVHumans
NeuS2 supports the data format provided by data_NeuS2.
.
<data_NeuS2>
|---images
| |---000000 # target frame of the scene
| | |---image_c_000_f_000000.png
| | |---image_c_001_f_000000.png
| | |---...
| |---000005
| | |---image_c_000_f_000000.png
| | |---image_c_001_f_000000.png
| | |---...
| |---...
|---train
| |---transform_000000.json
| |---transform_000005.json
| |---<transform_*.json>
|---test
| |---transform_000000.json
| |---transform_000005.json
| |---<transform_*.json>
|---transform_000000.json
|---transform_000005.json
|---<transform_*.json>
Run the Instant-ngp/torch-ngp on PKU-DyMVHumans
Instant-ngp supports the data format provided by data_ngp.
.
<data_ngp>
|---000000 # frame
| |---images
| | |---image_c_000_f_000000.png
| | |---image_c_001_f_000000.png
| | |---...
| |---transform_test.json
| |---transform_train.json
| |---transform_val.json
|---...
Run the Surface-SOS or 3D-Gaussian on PKU-DyMVHumans
Surface-SOS/3D-Gaussian supports the data format provided by data_COLMAP.
.
<data_COLMAP>
|---000000 # frame
| |---images
| | |---image_c_000_f_000000.png
| | |---image_c_001_f_000000.png
| | |---...
| |---masks
| | |---image_c_000_f_000000.png
| | |---image_c_001_f_000000.png
| | |---...
| |---sparse
| | |---cameras.bin
| | |---images.bin
| | |---points3D.bin
|---...
If you find this repo is helpful, please cite:
@article{zheng2024PKU-DyMVHumans,
title={PKU-DyMVHumans: A Multi-View Video Benchmark for High-Fidelity Dynamic Human Modeling},
author={Zheng, Xiaoyun and Liao, Liwei and Li,Xufeng and Jiao, Jianbo and Wang, Rongjie and Gao, Feng and Wang, Shiqi and Wang, Ronggang},
journal={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2024}
}
This repository is partly based on COLMAP, BackgroundMattingV2, NeuS, NeuS2, Instant-ngp, torch-ngp, 3D-Gaussian, and Surface-SOS.
We appreciate their contributions to the community.
Xiaoyun Zheng ([email protected])