Skip to content
This repository has been archived by the owner on Nov 6, 2019. It is now read-only.

Pass/total numbers inflated by "Test file" for testharness.js tests (harness status) #427

Closed
foolip opened this issue Feb 5, 2018 · 3 comments
Assignees

Comments

@foolip
Copy link
Member

foolip commented Feb 5, 2018

https://wpt.fyi/gyroscope is a directory where currently all tests except https://wpt.fyi/gyroscope/Gyroscope_insecure_context.html are actually failing, but the overall pass rates listed in https://wpt.fyi/ are 10/50.

This is because "Test file" counts as passing if there aren't any harness errors, for testharness.js tests. For test suites that are split into many files, this makes it harder to spot the "not supported at all" cases, and it would also inflate the 4/4 counts for #83. Example:
https://metrics5-dot-wptdashboard.appspot.com/metrics/gyroscope/Gyroscope-disabled-by-feature-policy.https.html

@mdittmer @lukebjerring FYI ^

Since "Test file" is the main and only result for reftests the solution would not be to just omit this line. And knowing about harness errors is useful, so omitting it for just testharness.js tests is also not a complete solution.

@foolip
Copy link
Member Author

foolip commented Feb 5, 2018

I think this might work:

  • Call it "Harness error" instead of "Test file" for testharness.js tests
  • Don't count that in any summary numbers
  • Include harness errors in the upcoming "anomalies" view.

@rwaldron
Copy link
Contributor

rwaldron commented Feb 8, 2018

This is being addressed in a redesign (which we've scheduled a presentation for Wed the 14th)

@foolip
Copy link
Member Author

foolip commented Apr 17, 2018

Moved to web-platform-tests/wpt.fyi#62.

@foolip foolip closed this as completed Apr 17, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants