-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
262 lines (194 loc) · 14.4 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
---
layout: default
---
<section style="text-align: justify">
<h1> Visual Domain Adaptation Challenge </h1>
<h2> (VisDA-2019) </h2>
</br>
[<a href = "#news">News</a>]
[<a href = "#overview">Overview</a>]
[<a href = "https://github.com/VisionLearningGroup/visda-2019-public">Data and Code</a>]
[<a href= "#prizes">Prizes</a>]
[<a href = "#evaluation">Evaluation</a>]
[<a href = "#rules">Rules</a>]
[<a href = "#faq">FAQ</a>]
[<a href = "https://sites.google.com/view/task-cv2019">TASK-CV Workshop</a>]
[<a href = "#organizers">Organizers</a>]
<!--[<a href = "#sponsors">Sponsors</a>]-->
</br> </br> </nr>
<a name = "news"></a>
<h2 class="section-title"> News </h2>
<p style="color:red"> Click <a href="https://ai.bu.edu/visda-2022/">[here]</a> to learn about our VisDA 2022 Challenge.</p>
<p>Introducing the 2019 VisDA Challenge! This year we are using a new <a href="http://ai.bu.edu/M3SDA/">[DomainNet dataset]</a> to promote <b>Multi-Source Domain Adaptation</b> and <b>Semi-Supervised Domain Adaptation</b> tasks. For details about last year's challenges and winning methods, see VisDA <a href = "http://ai.bu.edu/visda-2017/">[2017]</a> and <a href = "http://ai.bu.edu/visda-2018/">[2018]</a> pages.</p>
<p><b>Important Announcement:</b> in contrast with prior VisDA challenges, this year's challenge winners will be <b>required</b> to release a four-page technical report <b>and</b> code to replicate their results. See the corresponding <a href="#faq">rules</a> section for details. </p>
<ul>
<li><b>Oct 10</b> We released the labels for the test data! Use this <a href="http://ai.bu.edu/DomainNet/">link </a>to download the dataset.</li>
<li><b>Oct 10</b> We finalized challenge winners. Congratulations to the challenge winners!</li>
<li><b>Sep 26</b> We extend the deadline to Sep 28th, 11:59 am EST/ Sep 28th, 3:59 pm UTC </li>
<li><b>Sep 26</b> We updated the prize details, please check the <a href="#prizes">PRIZES</a> section for details.</li>
<li><b>Aug 30</b> We updated the rules, please check the <a href="#rules">RULES</a> section for details.</li>
<li><b>Aug 30</b> We have released the test data and the testing phase begins!</li>
<li><b>Aug 8</b> Ground Truth labels for VisDA 2017 released, see <a href="https://github.com/VisionLearningGroup/taskcv-2017-public/tree/master/classification">Git Repo</a> for details</li>
<li><b>July 3</b> Evaluation Server Online Now</li>
<li><b>April 9</b> Registration starts</li>
</ul>
</br>
<a name = "winners"></a>
<h2 class="section-title"> Winners of VisDA-2019 challenge </h2>
<div class="summary">
<b style="font-size: 20px;"> Multi-Source Domain Adaptation </b> <a href="https://competitions.codalab.org/competitions/20256#results"> [full leaderboard]</a>
</br> </br>
<table style="width:100%" class="table">
<tr class="active">
<th width='3%'>#</th>
<th width='25%'>Team Name</th>
<th>Affiliation</th>
<th>Score</th>
</tr>
<tr>
<td> 1 </td>
<td>VARMS</td>
<td>JD AI Research</td>
<td> 76.0 [<a href="">slides</a>] [<a href="http://csr.bu.edu/ftp/visda/2019/tech_reports/multi-source-1st.pdf">tech report</a>]</td>
</tr>
<tr>
<td> 2 </td>
<td>denemmy</td>
<td> Samsung AI Center Moscow</td>
<td>71.6 [<a href="">slides</a>] [<a href="http://csr.bu.edu/ftp/visda/2019/tech_reports/multi-source-2nd.pdf">tech report</a>]</td>
</tr>
<tr>
<td> 3 </td>
<td>SCALAR</td>
<td>Seoul National University, CV Lab<br>
AI Laboratory, LG Electronics</td>
<td>69.6 [<a href="">slides</a>] [<a href="http://csr.bu.edu/ftp/visda/2019/tech_reports/multi-source-3rd.pdf">tech report</a>]</td>
</tr>
</table>
</br>
<b style="font-size: 20px;"> Semi-Supervised Domain Adaptation </b> <a href="https://competitions.codalab.org/competitions/20257#results"> [full leaderboard]</a>
</br> </br>
<table style="width:100%" class="table">
<tr class="active">
<th width='3%'>#</th>
<th width='25%'>Team Name</th>
<th>Affiliation</th>
<th>Score</th>
</tr>
<tr>
<td> 1 </td>
<td>lunit</td>
<td>Lunit Inc</td>
<td>72.0 [<a href="">slides</a>] [<a href="http://csr.bu.edu/ftp/visda/2019/tech_reports/semi-supervised-1st.pdf">tech report</a>]</td>
</tr>
<tr>
<td> 2 </td>
<td>VARMS</td>
<td>JD AI Research</td>
<td>71.4 [<a href="">slides</a>] [<a href="http://csr.bu.edu/ftp/visda/2019/tech_reports/semi-supervised-2nd.pdf">tech report</a>]</td>
</tr>
<tr>
<td> 3 </td>
<td> denemmy </td>
<td>Samsung AI Center Moscow</td>
<td>71.3 [<a href="">slides</a>] [<a href="http://csr.bu.edu/ftp/visda/2019/tech_reports/semi-supervised-3rd.pdf">tech report</a>]</td>
</tr>
</table>
</br>
<a name = "overview"></a>
<h2 class="section-title"> Overview </h2>
<div class="summary">
<p> We are pleased to announce the 2019 Visual Domain Adaptation (VisDA2019) Challenge! It is well known that the success of machine learning methods on visual recognition tasks is highly dependent on access to large labeled datasets. Unfortunately, performance often drops significantly when the model is presented with data from a new deployment domain which it did not see in training, a problem known as <i>dataset shift</i>. The VisDA challenge aims to test domain adaptation methods’ ability to transfer source knowledge and adapt it to novel target domains. </p>
</br>
<p> The competition will take place during the months of June -- September 2019, and the top performing teams will be invited to present their results at the workshop at <a href="https://sites.google.com/view/task-cv2019">ICCV 2019</a> in Seoul, South Korea. This year’s challenge includes two tracks: </p>
<ul>
<li>Multi-Source Domain Adaptation</li> <li>Semi-Supervised Domain Adaptation</li>
</ul>
<p> on a new <a href="http://ai.bu.edu/M3SDA/">[DomainNet dataset]</a>. Participants are welcome to enter in one or both tracks. </p>
</br>
<figure>
<img class="img-responsive" style="dipslay:block;margin-left:auto;margin-right:auto" src="http://ai.bu.edu/M3SDA/imgs/data_examples.png"/>
<img class="img-responsive" style="dipslay:block;margin-left:auto;margin-right:auto" src="http://ai.bu.edu/M3SDA/imgs/statistics.png"/>
</figure>
</br>
</br>
<a name = "sponsors"></a>
<h2 class="section-title"> Sponsors </h2>
<img src="nvidia.jpg" height="120">
<img src="https://midas.bu.edu/images/logos/BU-logo.png" height="100">
</br>
</br>
<a name = "prizes"></a>
<h2 class="section-title"> Prizes </h2>
<p> The top three teams in each track will receive prizes:</p>
<ul>
<li>1st place: <a href = "https://www.nvidia.com/en-us/geforce/graphics-cards/rtx-2080-ti/"> NVIDIA GeForce RTX 2080 Ti GPU </a></li>
<li>2nd place: USD 500 Amazon Gift Card </li>
<li>3rd place: USD 250 Amazon Gift Card </li>
</ul>
</br>
<a name = "evaluation"></a>
<h2 class="section-title"> Evaluation </h2>
<p> We will use CodaLab to evaluate submissions and maintain a leaderboard. To register for the evaluation server, please create an account on <a href = "https://competitions.codalab.org/">CodaLab</a> and and enter as a participant in one of the following competitions: </p>
<ul>
<li><a href = "https://competitions.codalab.org/competitions/20256">Multi-Source Domain Adaptation</a></li>
<li><a href = "https://competitions.codalab.org/competitions/20257">Semi-Supervised Domain Adaptation</a></li>
</ul>
<p>If you are working as a team, you have the option to register for one account for your team or register multiple accounts under the same team name. If you choose to use one account, please indicate the names of all of the members on your team. This can be modified in the “User Settings” tab. If your team registers for multiple accounts, please do so using the protocol explained by CodaLab <a href = "https://github.com/codalab/codalab-competitions/wiki/User_Teams">here</a>. Regardless of whether you register for one or multiple accounts, your team must adhere to the per-team submission limits (20 entries per day per team during the validation phase). </p>
<p>Please refer to the instructions in the <a href = "https://github.com/VisionLearningGroup/visda-2019-public">DevKit</a> ReadMe file for specific details on submission formatting and evaluation for the classification and segmentation challenges.</p>
</br>
<a name = "rules"></a>
<h2 class="section-title"> Rules </h2>
<p>The VisDA challenge tests adaptation and model transfer, so the rules are different than most challenges. Please read them carefully.</p>
<p><b>Supervised Training:</b> Teams may only submit test results of models trained on the source domain data. To ensure equal comparison, we also do not allow any other external training data, modifying the provided training dataset, or any form of manual data labeling. </p>
<p><b>Unsupervised training:</b> Models can be adapted (trained) on the test data in an unsupervised way, i.e. without labels. </p>
<p><b>Source Models:</b> The performance of a domain adaptation algorithm greatly depends on the baseline performance of the model trained only on source data. We ask that teams submit two sets of results: 1) predictions obtained only with the source-trained model, and 2) predictions obtained with the adapted model. See the development kit for submission formatting details. </p>
<p><b>Leaderboard:</b>The main leaderboard for each competition track will show results of adapted models and will be used to determine the final team ranks. The expanded leaderboard will additionally show the team's source-only models, i.e. those trained only on the source domain without any adaptation. These results are useful for estimating how much the method improves upon its source-only model, but will not be used to determine team ranks. </p>
<p><b>Rank:</b>The final rank will be determined by the overall accuracy on <i>clipart</i> and <i>painting</i> domain. You can train two separate models for <i>clipart</i> and <i>painting</i>.</p>
</br>
<a name = "faq"></a>
<h2 class="section-title"> FAQ </h2>
<ol>
<li><b>What's the training setting for unsupervised multi-source domain adaptation?</b></li>
<p>Ideally, you should train two <b>identical</b> models with the same <b>strategy/approach/hyperparameter</b> on following settings:</p>
<p>Model I: Train on labeled images in sketch_train.txt (#49,115) /real_train.txt (#122,563) /quickdraw_train.txt (#120,750) /infograph_train.txt (#37,087) + unlabeled images in clipart_train.txt (#34,019), Test on unlabeled images in clipart_test.txt (#14,814). </p>
<p>Model II: Train on labeled images in sketch_train.txt (#49,115) /real_train.txt (#122,563) /quickdraw_train.txt (#120,750) /infograph_train.txt (#37,087) + unlabeled images in painting_train.txt (#52,867), Test on unlabeled images in painting_test.txt (#22,892). </p>
<p>The submission file should contain the predictions of model I on 14,814 testing images and the predictions of model II on 22,892 testing images. The final ranking will be determined by how many correct predictions do the submission file have within the (14,814+22,892) images.
One submission example can be found with the following link: <a href="https://github.com/VisionLearningGroup/visda-2019-public/blob/master/multisource/submission_example/result.txt"> submission_example.txt </a></p>
<li><b>Can we train models on data other than the source domain?</b></li>
<p> Participants may elect to pre-train their models only on ImageNet. Please refer to the challenge evaluation instructions found in the <a href = "https://github.com/VisionLearningGroup/visda-2019-public">DevKit</a> for more details. </p>
<li><b>In unsupervised multi-source domain adaptation challenge, can we use testing split of the given data to tune the parameter?</b></li>
<p>No, in the training phase, only the training split can be used to train the model. In other words, utilizing the testing split from the source domains or target domain is prohibited. </p>
<li><b>Can we assign pseudo labels to the unlabeled data in the target domain?</b></li>
<p>Yes, assigning pseudo labels is allowed as long as no human labeling is involved. </p>
<li><b>Do we have to use the provided baseline models?</b></li>
<p>No, these are provided for your convenience and are optional. </p>
<li><b>How many submissions can each team submit per competition track?</b></li>
<p>For the validation domain, the number of submissions per team is limited to 20 upload per day and there are no restrictions on total number of submissions. For the test domain, the number of submissions per team is limited to 1 upload per day and 20 uploads in total. Only one account per team must be used to submit results. Do not create multiple accounts for a single project to circumvent this limit, as this will result in disqualification. </p>
<li><b>Can multiple teams enter from the same research group?</b></li>
<p>Yes, so long as each team is comprised of different members.</p>
<li><b>Can external data be used?</b></li>
<p>The allowed training data consists of the VisDA 2019 Training set. Optional initialization of models with weights pre-trained on ImageNet is allowed and must be declared in the submission. Please see the <a href = "#rules">challenge rules</a> for more details. </p>
<li><b>Are challenge participants required to reveal all details of their methods?</b></li>
<p><b> Yes! </b> The top performing teams are required to include a four+ page write-up regarding their methods and code to reproduce their results to the claim the victory. The detailed procedure for releasing the code is to be determined. </p>
<li><b>Do participants need to adhere to TASK-CV abstract submission deadlines to participate in the challenge?</b></li>
<p>Submission of a <a href = "https://sites.google.com/view/task-cv2019">TASK-CV</a> workshop abstract is not mandatory to participate in the challenge; however, we request that any teams that wish to be considered for prizes or receive invitation to speak at the workshop submit a 4-page abstract. The top-performing teams that submit abstracts will be invited to present their approaches at the workshop. </p>
</br>
</br>
<a name = "organizers"></a>
<h2 class="section-title"> Organizers </h2>
Kate Saenko (Boston University),
Xingchao Peng (Boston University),
Ben Usman (Boston University),
Kuniaki Saito (Boston University),
Ping Hu (Boston University),
</br>
</br>
<a name = "acknowledgement"></a>
<h2 class="section-title"> Acknowledgement </h2>
We thank Siddharth Mysore Sthaneshwar, Andrea Burns, Reuben Tan, Yichen Li for testing the evaluation server and the website.
</br>
<div height=1000px> </div>
</div><!--//summary-->
</section><!--//section-->