Bodyfitting is the SMPL fitting tool in "Generalizable Neural Performer: Learning Robust Radiance Fields for Human Novel View Synthesis" and GeneBody Dataset.
This toolbox can register SMPL from calibrated motion capture images, as well as the synthetic meshes; SMPL+D and texture fitting is also provided.
The project is built on python3.6 and torch 1.2, you can set up the environment as:
conda env create -n bodyfitting python=3.6
pip install -r requirements.txt
cd thirdparty/neural_renderer && python setup.py install
cd thirdparty/mesh_grid && python setup.py install
Note: neural_renderer in this repo is a modified version for texture fitting. mesh_grid is our implemetation of mesh closest point.
Please download the SMPL/SMPLx model, and HMR model from here, and put them in the data folder.
We require Openpose for 2D keypoint detection, please build the cpp version from the instructions.
Other alts such as MMPose also fits this framework, as long as appropriate joint mapping is assigned.
Take fitt GeneBody and RenderPeople as an example, first you should organize your data directory as follows
├──genebody/
├──amanda/
├──barry/
├──...
├──rp_scans/
├──rp_alexandra_posed_013_OBJ/
├──rp_alisha_posed_001_OBJ/
├──rp_alvin_posed_006_OBJ/
├──...
You can fit your motion capture data using the following command, choose smpl_type
and use_mask
to enable silhouette fitting if human segmentation is given.
For example to fit amanda
sequnece
python apps/genebody_fitting.py --target_dir path_to_genebody --subject amanda --openpose_dir path_to_openpose --output_dir path_to_output --smpl_type smplx --use_mask --tasks openpose smplify output
You can find the SMPL/SMPLx obj file and optimized parameters in path_to_output.
You can fit your Render People mesh data using the following command. To perform SMPL+D and texture fitting, please add smpld
and texfit
in tasks list.
python apps/rp_fitting.py --target_dir path_to_rp_scans --openpose_dir path_to_openpose --output_dir path_to_output --smpl_type smplx --smpl_uv_dir ./smpl_uv --tasks openpose smplify smpld texfit output
Texture fitting takes opitmized SMPL+D as input, and optimize the texture image by the L2 loss between rendered images of groudtruth mesh and SMPL+D mesh. The process and results are as follows:
Left: Texture optimization process, Right: Comparison of ground truth mesh and textured SMPL+D
If you find this repo useful for your work, please cite the follow technical paper
@article{cheng2022generalizable,
title={Generalizable Neural Performer: Learning Robust Radiance Fields for Human Novel View Synthesis},
author={Cheng, Wei and Xu, Su and Piao, Jingtan and Qian, Chen and Wu, Wayne and Lin, Kwan-Yee and Li, Hongsheng},
journal={arXiv preprint arXiv:2204.11798},
year={2022}
}
@inproceedings{SMPL-X:2019,
title = {Expressive Body Capture: {3D} Hands, Face, and Body from a single Image},
author = {Pavlakos, Georgios and Choutas, Vasileios and Ghorbani, Nima and Bolkart, Timo and Osman, Ahmed A. A. and Tzionas, Dimitrios and Black, Michael J.},
booktitle = {Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
pages = {10975--10985},
year = {2019}
}