Experimental Tag for benchmarks

[nv-rborkar]

Agile tag is a way for MLPerf to stay agile & make early bets on upcoming hot benchmarks.
It allows adopting viral benchmarks while also providing a way to update, refresh or tweak them as the ML landscape changes quickly instead of getting locked-in to a 2 year cadence.

Ideally all benchmarks should have the agility to refresh if landscape warrants faster change e.g. update the sequence length of existing LLM models or tweak outdated architectures but we should also balance the churn to reduce submitter burden and prolong investment !/$.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[nv-rborkar]

4/25: Training WG agrees with this proposal

[itayhubara]

Two comments:

1. I cannot find the "expected lifetime of 4 rounds" rule anywhere - @nv-rborkar can you please point me to that line
2. I believe that changing the rule to allow one agile (instead of two) per round should be enough (we don't usually have two "real agile" models that we wish to add per year)

[hiwotadese]

WG agreed on 06/20 to add "Benchmarks live for a minimum of 2 years. Early retirement requires formal WG vote" to the rules to handle early retirement.

[ShriyaPalsamudram]

#181 - PR to address benchmark lifecycle and early retirement process.