-
Notifications
You must be signed in to change notification settings - Fork 17
/
index.html
executable file
·239 lines (201 loc) · 11.6 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3c.org/TR/1999/REC-html401-19991224/loose.dtd">
<html xml:lang="en" xmlns="http://www.w3.org/1999/xhtml" lang="en"><head>
<title>face-vid2vid</title>
<meta http-equiv="Content-Type" content="text/html; charset=windows-1252">
<meta property="og:image" content="web_gifs/teaser.gif"/>
<meta property="og:title" content="One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing" />
<script src="lib.js" type="text/javascript"></script>
<script src="popup.js" type="text/javascript"></script>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.1.0/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/vue/2.2.4/vue.min.js"></script>
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-53682931-1', 'auto');
ga('send', 'pageview');
</script>
<script type="text/javascript">
// redefining default features
var _POPUP_FEATURES = 'width=500,height=300,resizable=1,scrollbars=1,titlebar=1,status=1';
</script>
<link media="all" href="glab.css" type="text/css" rel="StyleSheet">
<style type="text/css" media="all">
IMG {
PADDING-RIGHT: 0px;
PADDING-LEFT: 0px;
FLOAT: right;
PADDING-BOTTOM: 0px;
PADDING-TOP: 0px
}
#primarycontent {
MARGIN-LEFT: auto; ; WIDTH: expression(document.body.clientWidth >
1150? "1150px": "auto" ); MARGIN-RIGHT: auto; TEXT-ALIGN: left; max-width:
1150px }
BODY {
TEXT-ALIGN: center
}
</style>
<meta content="MSHTML 6.00.2800.1400" name="GENERATOR"><script src="b5m.js" id="b5mmain" type="text/javascript"></script></head>
<body>
<div id="primarycontent">
<center><h1>One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing</h1></center>
<center><h3>
<a href="https://tcwang0509.github.io/">Ting-Chun Wang</a>
<a href="https://arunmallya.github.io//">Arun Mallya</a>
<a href="http://mingyuliu.net/">Ming-Yu Liu</a>
</center>
<center><h3>NVIDIA Corporation</h3></center>
<center><h4>CVPR 2021 (Oral)</h4></center>
<center><h3><strong>
<a href="main.pdf">[Paper]</a>
<!-- <a href="https://arxiv.org/abs/2011.15126">[arXiv]</a> -->
<a href="https://github.com/tcwang0509/TalkingHead-1KH">[Dataset]</a>
<!-- <a href="demo.html">[Demo]</a> -->
<a href="https://youtu.be/nLYg9Waw72U">[Video]</a>
<a href="https://www.youtube.com/watch?v=smrcnZ5Eg4A">[Talk]</a>
</strong> </h3></center>
<center><a href="web_gifs/teaser.gif">
<img style="PADDING-LEFT: 0px; PADDING-RIGHT: 40px;" src="web_gifs/teaser.gif" width="1024"> </a></center>
<h2>Abstract</h2>
<div style="font-size:14px"><p>We propose a neural talking-head video synthesis model and demonstrate its application to video conferencing. Our model learns to synthesize a talking-head video using a source image containing the target person's appearance and a driving video that dictates the motion in the output. Our motion is encoded based on a novel keypoint representation, where the identity-specific and motion-related information is decomposed unsupervisedly. Extensive experimental validation shows that our model outperforms competing methods on benchmark datasets. Moreover, our compact keypoint representation enables a video conferencing system that achieves the same visual quality as the commercial H.264 standard while only using one-tenth of the bandwidth. Besides, we show our keypoint representation allows the user to rotate the head during synthesis, which is useful for simulating a face-to-face video conferencing experience.</p></div>
<a href="https://arxiv.org/abs/2011.15126"><img style="float: left; padding: 10px; PADDING-RIGHT: 30px;" alt="paper thumbnail" src="images/paper_thumbnail.png" width=170></a>
<br>
<h2>Paper</h2>
<p><a href="https://arxiv.org/pdf/2011.15126">arXiv</a> </p>
<h2>Citation</h2>
<p>Ting-Chun Wang, Arun Mallya, Ming-Yu Liu. "One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing."<br>
In CVPR, 2021.
<a href="Bibtex.txt">Bibtex</a>
</p><br>
<h2>Code</h2>
<p>This work is based upon <a href="https://github.com/NVlabs/imaginaire">Imaginaire</a> for non-commercial use.
For business inquiries, please visit our website and submit the form: <a href="https://www.nvidia.com/en-us/research/inquiries">NVIDIA Research Licensing</a>.
<br>
<h2>Dataset</h2>
<p>The dataset we collected to train the model can be downloaded <a href="https://github.com/tcwang0509/TalkingHead-1KH">here</a>. Note that the number of videos differ from that in the paper because a different preprocessing script was used to split the videos.
<br>
<h2>Demo</h2>
Please visit <a href="demo.html">this page</a> for an online demo.
<h1 align='center'>Our Example Results</h1>
<table align="center" border="0" cellspacing="0" cellpadding="0">
<tr>
<td align="center" valign="middle">
<p> <iframe width="1024" height="575" src="https://www.youtube.com/embed/nLYg9Waw72U" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe></p>
</td>
</tr>
</table>
<p> </p>
<!-- <h1 align='center'>Online Demo</h1>
<p>To run the demo, you can either use the default images or upload your own image. Please do the following steps.</p>
<ol>
<li>If you use the default images, simply click on one of <b>Image</b> buttons.</li>
<li>If you choose to upload your own image, please first click <b>Choose File</b> then click <b>Upload</b>. Then plase draw a rectangle to cover the head region over the image. Simply left-click the mouse, drag, and release.</li>
<li>Specify the desired rotation angles by scrolling the bars.</li>
<li>Click on <b>Rotate</b>. The result should appear on the right.</li>
</ol>
<form oninput="pitch_result.value=parseInt(pitch.value);yaw_result.value=parseInt(yaw.value);roll_result.value=parseInt(roll.value)">
<div class="slidecontainer">
<input type="range" min="-30" max="30" value="0" class="slider" id="pitch" oninput='changePitch();'>
pitch: <output name="pitch_result" for="pitch">0</output>
</div>
<div class="slidecontainer">
<input type="range" min="-30" max="30" value="0" class="slider" id="yaw" oninput='changeYaw();'>
yaw: <output name="yaw_result" for="yaw">0</output>
</div>
<div class="slidecontainer">
<input type="range" min="-30" max="30" value="0" class="slider" id="roll" oninput='changeRoll();'>
roll: <output name="roll_result" for="roll">0</output>
</div>
</form>
<form action='#' onsubmit="return false;">
<input type='button' class="w3-button w3-gray" id='exp_img1' value='Image 1' onclick='displayImage(0);' />
<input type='button' class="w3-button w3-gray" id='exp_img2' value='Image 2' onclick='displayImage(1);' />
<input type='button' class="w3-button w3-gray" id='exp_img3' value='Image 3' onclick='displayImage(2);' />
<input type='button' class="w3-button w3-gray" id='exp_img4' value='Image 4' onclick='displayImage(3);' />
<input type='button' class="w3-button w3-gray" id='exp_img5' value='Image 5' onclick='displayImage(4);' />
<input type='button' class="w3-button w3-gray" id='exp_img6' value='Image 6' onclick='displayImage(5);' />
<input type='button' class="w3-button w3-gray" id='exp_img7' value='Image 7' onclick='displayImage(6);' />
<input type='button' class="w3-button w3-gray" id='exp_img8' value='Image 8' onclick='displayImage(7);' />
<button id="mine2others" class="w3-button w3-red" style="background-color:lightgreen">Rotate</button>
</form>
<form action='#' onsubmit="return false;">
<input type='file' class="w3-button w3-red" id='imgfile' />
<input type='button' class="w3-button w3-yellow" id='btnLoad' value='Upload' style="background-color:yellow" onclick='loadImage();' />
</form>
<p class="w3-gray">By uploading a file, you are agreeing to this service <a href="term.txt">Terms and Conditions</a>. See our <a href="https://www.nvidia.com/en-us/about-nvidia/privacy-policy/">Privacy Policy</a>.</p>
<canvas id="viewport_default"></canvas>
<canvas id="output_default"></canvas>
<canvas id="viewport"></canvas>
<canvas id="output"></canvas>
<script src="face_redirection.js"> </script> -->
<h1 align='center'>Video Reconstruction</h1>
<table border="0" cellspacing="10" cellpadding="0">
<tr>
<td align="center" valign="middle">
<p> <iframe width="560" height="315" src="https://www.youtube.com/embed/xqJ2tSL2mHs" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe></p>
</td>
<td align="center" valign="middle">
<p> <iframe width="560" height="315" src="https://www.youtube.com/embed/tDQ_6cO4qSc" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe></p>
</td>
</tr>
</table>
<p> </p>
<h1 align='center'>Head Rotation</h1>
<table border="0" cellspacing="10" cellpadding="0">
<tr>
<td align="center" valign="middle">
<p> <iframe width="560" height="315" src="https://www.youtube.com/embed/dGY8NbG11Ng" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe></p>
</td>
<td align="center" valign="middle">
<p> <iframe width="560" height="315" src="https://www.youtube.com/embed/fjzobpA0Sgs" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe></p>
</td>
</tr>
</table>
<p> </p>
<h1 align='center'>Face Frontalization</h1>
<table border="0" cellspacing="10" cellpadding="0">
<tr>
<td align="center" valign="middle">
<p> <iframe width="560" height="315" src="https://www.youtube.com/embed/OFStV3VRgAI" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe></p>
</td>
<td align="center" valign="middle">
<p> <iframe width="560" height="315" src="https://www.youtube.com/embed/7QQ7t2z_sfI" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe></p>
</td>
</tr>
</table>
<p> </p>
<h1 align='center'>Motion Transfer</h1>
<table border="0" cellspacing="10" cellpadding="0">
<tr>
<td align="center" valign="middle">
<p> <iframe width="560" height="315" src="https://www.youtube.com/embed/lSLsiEmfEtk" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe></p>
</td>
<td align="center" valign="middle">
<p> <iframe width="560" height="315" src="https://www.youtube.com/embed/wOc4PQCJGXg" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe></p>
</td>
</tr>
</table>
<p> </p>
<h2>Citation</h2>
<p>If you find this useful for your research, please use the following.<br>
@inproceedings{wang2021facevid2vid,<br>
  title={One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing},<br>
  author={Ting-Chun Wang and Arun Mallya and Ming-Yu Liu}, <br>
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}, <br>
  year={2021} <br>
}</p>
<div style="display:none">
<!-- GoStats JavaScript Based Code -->
<script type="text/javascript" src="http://gostats.com/js/counter.js"></script>
<script type="text/javascript">_gos='c3.gostats.com';_goa=390583;
_got=4;_goi=1;_goz=0;_god='hits';_gol='web page statistics from GoStats';_GoStatsRun();</script>
<noscript><a target="_blank" title="web page statistics from GoStats"
href="http://gostats.com"><img alt="web page statistics from GoStats"
src="http://c3.gostats.com/bin/count/a_390583/t_4/i_1/z_0/show_hits/counter.png"
style="border-width:0" /></a></noscript>
</div>
<!-- End GoStats JavaScript Based Code -->
</body></html
>