Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Also ignore ERROR in slow test check #13158

Merged
merged 1 commit into from
Sep 24, 2018
Merged

Also ignore ERROR in slow test check #13158

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions tools/wptrunner/wptrunner/stability.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,8 +115,9 @@ def find_slow_status(test):
"""Check if a single test almost times out.

We are interested in tests that almost time out (i.e. likely to be flaky).
Therefore, timeout statuses are ignored, including (EXTERNAL-)TIMEOUT &
CRASH (tests that time out may be marked as CRASH if crashes are detected).
Therefore, timeout statuses are ignored, including (EXTERNAL-)TIMEOUT.
CRASH & ERROR are also ignored because the they override TIMEOUT; a test
that both crashes and times out is marked as CRASH, so it won't be flaky.

Returns:
A result status produced by a run that almost times out; None, if no
Expand All @@ -125,7 +126,7 @@ def find_slow_status(test):
if "timeout" not in test:
return None
threshold = test["timeout"] * FLAKY_THRESHOLD
for status in ['PASS', 'FAIL', 'OK', 'ERROR']:
for status in ['PASS', 'FAIL', 'OK']:
if (status in test["longest_duration"] and
test["longest_duration"][status] > threshold):
return status
Expand Down
3 changes: 3 additions & 0 deletions tools/wptrunner/wptrunner/tests/test_stability.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@ def test_find_slow_status():
assert stability.find_slow_status({
"longest_duration": {"CRASH": 10},
"timeout": 10}) is None
assert stability.find_slow_status({
"longest_duration": {"ERROR": 10},
"timeout": 10}) is None
assert stability.find_slow_status({
"longest_duration": {"PASS": 1},
"timeout": 10}) is None
Expand Down