Skip to content

Organizer_Leaderboard

Jump to bottom
Adrien Pavão edited this page Dec 8, 2023 · 11 revisions

Contents

Leaderboard nuts and bolts

The Codalab leaderboard offers many features to display competition results. It can be configured to display results on multiple datasets, show execution time, show rank on all scores, etc. etc. This page will help you out... To view it, click "Results":

1. Leaderboard format.

In the YAML file, you can specify the leaderboard format. This is best explained via an example from the AutoML challenge:

leaderboard:
    leaderboards:
        Results: &RESULTS
            label: RESULTS                      # This will be the title of the table
            rank: 1
    columns:
        set1_score:                             # This corresponds to 1st value provided by the scoring program
            leaderboard: *RESULTS
            label: Set 1                        # This is the name of the column in the table
            numeric_format: 4                   # This is the number of decimals
            rank: 2                             # This is the number of the column (column 2)
        set2_score:                             # This corresponds to 2nd value provided by the scoring program
            leaderboard: *RESULTS
            label: Set 2
            numeric_format: 4
            rank: 3
        set3_score:                             # This corresponds to 3rd value provided by the scoring program
            leaderboard: *RESULTS
            label: Set 3
            numeric_format: 4
            rank: 4
        set4_score:                             # This corresponds to 4th value provided by the scoring program
            leaderboard: *RESULTS
            label: Set 4
            numeric_format: 4
            rank: 5
        set5_score:                             # This corresponds to 5th value provided by the scoring program
            leaderboard: *RESULTS
            label: Set 5
            numeric_format: 4
            rank: 6
        ave_score:                              # We will show the average rank among several scores
            leaderboard: *RESULTS
            label: <Rank>                       # This is the column name in the table
            numeric_format: 4
            rank: 1                             # This will be the 1st column
            computed:                           # The scores that will be used to compute average ranking
                operation: Avg
                fields: set1_score, set2_score, set3_score, set4_score, set5_score
        Duration:                               # We will also display duration (in seconds)
            leaderboard: *RESULTS
            label: Duration 
            numeric_format: 2
            rank: 7

The leaderboard will display the results of the file score.txt, which is returned by the scoring program, and has the following format:

set1_score: 0.813519760093
set2_score: 0.745929379262
set3_score: 0.523620210384
set4_score: 0.793769274565
set5_score: 0.692495854273
Duration: 12

The leaderboard does not show all the decimals, it will limit itself to the number of decimals specified in the numeric_format. This is how this will look like:

You can see the average rank among the various tasks in the <Rank> column.

Note that you can also compute the weighted average rank by adding a weights field in the YAML file as following:

ave_score:
    leaderboard: *RESULTS
    label: < Rank >
    rank: 1
    sort: asc
    computed:
        operation: Avg
        fields: set1_score, set2_score, set3_score, set4_score, set5_score
        weights: 0.1, 0.4, 0.2, 0.2, 0.1

The ranking is then computed like this: set1_score_rank * 0.1 + set2_score_rank * 0.4 + ....

/!\ With ascending sorting (lower is better, the usual ranking), small coefficients means a more important task! /!\

Other possible operations: "Max" and "MRR" (mean reciprocal rank).

2. Leaderboard features

Several features can be enabled via the YAML file or the editor. To enter the editor, go to your competition and click "Edit": The features related to the leaderboard include:

  • force_submission_to_leaderboard: <True/False. If false, participants must submit manually. Otherwise, normally the last submission made is the one displayed on the leaderboard.>
  • If submission beats old score, put submission on leaderboard: Show best score instead of last score (look for this box under phases: each phase can turn this feature on/off).
  • disallow_leaderboard_modifying: <True/False. If true, submissions made to the leaderboard cannot be changed by the participants.>
  • Anonymous leaderboard: If checked, participants' names will not appear in the leaderboard.
  • auto_migration: <True /False. First submission = last one made in the previous phase.> See details.
  • Leaderboard Mode: . The mode "Hide Results" makes the leaderboard results invisible to non organizers (makes the leaderboard "private"). Useful when the organizers want to review results in the final phase before revealing them publicly.
  • enable_detailed_results: <True/False. True if and HTML file is written:> The scoring program may optionally also return a file scores.html containing detailed results (in free format), which will be linked from the leaderboard in the column "Detailed Results". This will only work if "enable_detailed_results" is set to True. See Detailed Results Page for details.
  • selection default: When selection default is > 0 the columns will be sorted in descending order when displayed on the Codalab website.

Public submissions

There is an option that can be turned on to activate a second tab next to the leaderboard, which makes available to everybody the submissions of the participants, if they choose to share them:

To enable it in the editor, ckeck the box: Allow sharing of public submissions. If checked, participants will be able to share their submissions with others. This is how it will look like:

Clone this wiki locally