train.md

Training Instructions

Throughout the documentation we refer to VIBE root folder as $ROOT.

Data Preparation

During training, VIBE uses precomputed image features to reduce training time. Thus, we process the datasets into a standard format before using them for training. To obtain these standard training files, you need to run:

source scripts/prepare_training_data.sh

This script will first create a folder for the dataset files $ROOT/data/vibe_db, then process each dataset and save output files to this directory. Before proceeding, you need to download each of the datasets listed below, then modify the --dir argument in the script to point the directory of each dataset.

Datasets

• AMASS (https://amass.is.tue.mpg.de)

Directory structure:

amass
|-- ACCAD
|-- BioMotionLab_NTroje
|-- CMU
|-- ...
`-- Transitions_mocap

• InstaVariety

For your convenience, we uploaded the preprocessed InstaVariety data here (size: 18 GB). After downloading the file, put it under $ROOT/data/vibe_db. Do not forget to verify checksum for sanity check:

md5sum    : 8ec335d1d48bd54687ad5c9a6eeb2999
sha256sum : 7eadff77043cd85b49cbba8bfc9111c4305792ca64da1b92fb40fa702689dfa9

You may also preprocess the dataset yourself by downloading the preprocessed tfrecords provided by the authors of Temporal HMR.

Directory structure:

insta_variety
|-- train
|   |-- insta_variety_00_copy00_hmr_noS5.ckpt-642561.tfrecord
|   |-- insta_variety_01_copy00_hmr_noS5.ckpt-642561.tfrecord
|   `-- ...
`-- test
    |-- insta_variety_00_copy00_hmr_noS5.ckpt-642561.tfrecord
    |-- insta_variety_01_copy00_hmr_noS5.ckpt-642561.tfrecord
    `-- ...

• MPI-3D-HP (http://gvv.mpi-inf.mpg.de/3dhp-dataset)

Donwload the dataset using the bash script provided by the authors. We will be using standard cameras only, so wall and ceiling cameras aren't needed. Then, run this script to extract frames of videos.

Directory structure:

mpi_inf_3dhp
|-- S1
|   |-- Seq1
|   |-- Seq2
|-- S2
|   |-- Seq1
|   |-- Seq2
|-- ...
`-- util

• 3DPW (https://virtualhumans.mpi-inf.mpg.de/3DPW)

Directory structure:

3dpw
|-- imageFiles
|   |-- courtyard_arguing_00
|   |-- courtyard_backpack_00
|   |-- ...
`-- sequenceFiles
    |-- test
    |-- train
    `-- validation

• PennAction (http://dreamdragon.github.io/PennAction/)

Directory structure:

pennaction
|-- frames
|   |-- 0000
|   |-- 0001
|   |-- ...
`-- labels
    |-- 0000.mat
    |-- 0001.mat
    `-- ...

• PoseTrack (https://posetrack.net/)

Directory structure:

posetrack
|-- images
|   |-- train
|   |-- val
|   |-- test
`-- posetrack_data
    `-- annotations
        |-- train
        |-- val
        `-- test

Training

Run the command below to start training.

python train.py --cfg configs/config.yaml

See configs/config.yaml or config.py to play with different configurations.