Perceptual Conversational Head Generation with Regularized Driver and Enhanced Renderer

Zhewei Huang, Ailin Huang, Shuchang Zhou

YouTube | arXiv

Introduction

This project is the implement of Perceptual Conversational Head Generation with Regularized Driver and Enhanced Renderer. We ranked first place in the listening head generation track and second place in the talking head generation track in the official ranking of MM2022-ViCo Conversational Head Generation Challenge. Our team name is Megvii_goodjuice.

The whole pipeline of challenge can be found on vico challenge baseline. We currently provide our major modification of the baseline.

Contributors: @P2Oileen, @hzwer

Modification

• Architecture & Pipeline

Clone vico challenge baseline, replace vico_challenge_baseline/vico/networks/audio_to_face.py, vico_challenge_baseline/vico/networks/speaker_generator.py, vico_challenge_baseline/PIRender/inference.py, and move u2net.py to vico_challenge_baseline/PIRender/u2net.py.

• Fusion

Use U2Net to segment the backgrounds and fuse them to remain the background unchanged. You should also download U2Net weights u2net_human_seg.pth from Google Drive and save in vico_challenge_baseline/PIRender/u2net_human_seg.pth.

• Image Boundary Inpainting

Change the padding mode of grid_sample in PIRenderer from "zeros" to "border".

Citation

@InProceedings{huang2022perceptual,
  title={Perceptual Conversational Head Generation with Regularized Driver and Enhanced Renderer},
  author={Huang, Ailin and Huang, Zhewei and Zhou, Shuchang},
  booktitle={Proceedings of the 30th ACM International Conference on Multimedia (MM'22)},
  year={2022}
}

The ViCo baseline method: 
@InProceedings{zhou2022responsive,
    title={Responsive Listening Head Generation: A Benchmark Dataset and Baseline},
    author={Zhou, Mohan and Bai, Yalong and Zhang, Wei and Yao, Ting and Zhao, Tiejun and Mei, Tao},
    booktitle={Proceedings of the European conference on computer vision (ECCV)},
    year={2022}
}