Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update mobile_inference_rules.adoc #28

Merged
merged 1 commit into from
Aug 9, 2023
Merged

Update mobile_inference_rules.adoc #28

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions rules/mobile_inference_rules.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,12 +80,22 @@ but excludes rooted devices.

If you are measuring the performance of an experimental framework or system, you
must make the system and framework you use available upon demand for
replication by MLCommons. This class of systems will be called RDI (research,
development, internal). RDI systems include reference hardware and software
replication by MLCommons. This class of systems will be called R&D SoC. R&D SoC systems include reference hardware and software
modifications such as rooted phones.

R&D SoC will be considered as any physical device including SoC under test, where the device is not commercially available, and the SoC under test is currently not commercially available, but can be made commercially available at some point in the future.

R&D SoC submitter should submit that SoC under Available category with a commercially available device in the next official submission after the commercial device is available.

If submitter fails to do this submission, original R&D SoC submission will be marked as invalid.
The R&D SoC can be optionally integrated, depending on the vendor’s willingness, into the backend of the MLPerf app. Any other member can verify this integration giving credibility to this system.

In order to be published R&D SoC submissions will be audited with the backend code and the respective logs by examining any modifications to the backend code integrated in the MLPerf app. R&D SoC submitters can also optionally send their engineering R&D device to MLCommons to reproduce their results

R&D SoC submitter should submit that SoC under Available category with a commercially available device in the next official submission after the commercial device is available

For Available Systems, devices used for testing should be publicly available as
commercial devices and device rooting is not allowed. For RDI Systesm, hardware and
commercial devices and device rooting is not allowed. For R&D SoC Systems, hardware and
software changes such as rooting should be explicitly mentioned in the device name.

=== Benchmark implementations must be shared
Expand Down
Loading