Home

Documentation

Welcome to the Codabench wiki!

Codabench is a platform allowing you to flexibly specify a benchmark. First you define tasks, e.g. datasets and metrics of success, then you specify the API for submissions of code (algorithms), add some documentation pages, and "CLICK!" your benchmark is created, ready to accept submissions of new algorithms. Participant results get appended to an ever-growing leaderboard.

You may also create inverted benchmarks in which the role of datasets and algorithms are swapped. You specify reference algorithms and your participants submit datasets.

Table of contents

1. Participants
2. Organizers
3. Developers and administrators
4. How you can contribute
5. How to contact us

1. Participants

• List of Current Benchmarks and Competitions
• Participating in a Competition

2. Organizers

Benchmark creation

• Getting Started Tutorial
• Advanced Tutorial
• How to transition from CodaLab to Codabench?
• Competition Creation: Form
• Competition Creation: Bundle
• Competition YAML Structure
• Competition Bundle Structure
• Competition Docker image
• Dataset competition creation and participate instruction
• Leaderboard Features
• Competition Examples
• Example: Cancer Benchmarks

Running a benchmark

• Benchmark Management & List Page
• Benchmark Detail Page and organizer features (manage participants, submissions, dumps...)
• Resource Management: Submissions, Datasets/Programs, Tasks and Competition Bundles (Resources interface)
• Queue Management
• Compute Worker Management & Setup
• Server Status (monitor the submissions at the compute workers level)

3. Developers and administrators

Codabench Platform Technical Information and Management

• Codabench Basic Installation Guide
• How to Deploy Server
• Administrator procedures
• Codabench Docker Architecture
• Submission Docker Container Layout
• Backups: Creating, Automating, and Restoring
• Submission Process Overview
• Robot submissions
• Running tests
• Automation
• Manual validation
• Validation and deployment of pull requests
• Upgrading Codabench

4. How you can contribute

• Use Codabench by either participating in a competition or hosting a new competition.
• Find a bug? Got a feature request? Submit a GitHub issue.
• P1 issues are the most important ones.
• Submit pull requests on GitHub to implement new features or fix bugs (see contributing.md).
• Improve this documentation.
• Let others know about Codabench!

5. How to contact us

• The preferred way is via posting a GitHub issue.
• In case of emergency, contact us directly.
• If you wish to get in touch with the community, you can use the Google Groups.