Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Review "Pass %" metric and consider a new "Pipeline %" metric #752

Open
smlambert opened this issue Dec 5, 2022 · 0 comments
Open

Review "Pass %" metric and consider a new "Pipeline %" metric #752

smlambert opened this issue Dec 5, 2022 · 0 comments

Comments

@smlambert
Copy link
Contributor

Pass percentage is calculated by Passed/Executed and does not currently take into account aborted tests. While we originally wanted Test Results Summary Service (TRSS) to summarize the results of tests that actually ran, it is likely more useful to include info about Aborted tests (which is often an indication of Infra issues).

For example, in the past weekend pipeline runs, most test jobs had to abort because there were no machines found online, so 7 of the 11 platforms could not launch AQA tests, due to

[Pipeline] nodesByLabel
13:19:17  Could not find any nodes with 'ci.role.test&&hw.arch.x86&&sw.os.windows&&sw.tool.docker' label
[Pipeline] timeout
13:19:17  Timeout expired 0 ms ago
13:19:17  Cancelling nested steps due to timeout
...
Finished: ABORTED

Screen Shot 2022-12-05 at 10 40 38 AM

The pass % shows to be 96.78% which are of the tests that did launch and could be run. The overall health of the weekly pipeline was poor, so it may be good to either change how Pass% is calculated, or figure out a calculation for a new metric, something called Pipeline% which includes "Aborted" jobs in the final metric. Pipeline % could measure the number of jobs in the pipeline that could run over the number of jobs that were launched.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant