New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add tests for scoring code #624

Open

priyakasimbeg opened this issue Feb 1, 2024 · 0 comments

Contributor

priyakasimbeg commented Feb 1, 2024 •

edited

Loading

Add unit and integration tests to test the following requirements:

In both strict=False and strict=True, to receive a finite score for a workload a submission must:

Reach the validation target on the fixed workload within the maximum runtime.
Reach the validation target fixed workload within 4x of the fastest submission.

In strict=True, to receive a finite score for a workload a submission must:

Reach the validation target for at least 3/5 studies (the median).
Take the best over 5 trials per study.
Reach the validation target on the held-out workload (corresponding to the fixed workload) within the maximum runtime.
Reach the validation target on the held-out workload (corresponding to the fixed workload) within 4x of the fastest submission.

priyakasimbeg mentioned this issue

Scoring code needs to account for held-out workload performance #588

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment