Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filter (all tests) by failure message #462

Open
foolip opened this issue Aug 21, 2018 · 6 comments
Open

Filter (all tests) by failure message #462

foolip opened this issue Aug 21, 2018 · 6 comments
Labels
enhancement New feature or request

Comments

@foolip
Copy link
Member

foolip commented Aug 21, 2018

In https://adam.orphis.net/~orphis/wpt.html @Orphis built in the ability to filter tests by failure messages, as "Output filter". wpt.fyi should be able to do this, for example searching for "message:TypeError" (strawman!) at the root level should filter to tests where that shows up in the output, such as:
https://wpt.fyi/results/webrtc/RTCRtpParameters-codecs.html?sha=6aef6676d2&label=stable (in Chrome + Edge)

This issue is very similar to #81, but that is about making search work on the results page for a specific tests, while this is about searching for tests that fail in a certain way.

@foolip foolip added the enhancement New feature or request label Aug 21, 2018
@lukebjerring
Copy link
Contributor

@mdittmer - can you confirm whether the searchcache includes an index for failure messages?

@foolip
Copy link
Member Author

foolip commented Feb 5, 2019

I can't say for certain, but in multiple discussions with @mdittmer I recall the answer being no.

@Hexcles
Copy link
Member

Hexcles commented Feb 12, 2019

No. And it'd either increase our memory requirement by several magnitudes or decrease the number of runs in the cache by several magnitudes.

@foolip
Copy link
Member Author

foolip commented Feb 15, 2019

FWIW, failure messages were included in my experiment fitting all runs into a few GB of memory.

@foolip
Copy link
Member Author

foolip commented Nov 6, 2019

This would have come in handy in web-platform-tests/wpt#20036 (comment).

@foolip
Copy link
Member Author

foolip commented Nov 6, 2019

Actually, it's probably about once a week that I find I'd want this but because I know it doesn't work I just move on. Another case today was web-platform-tests/wpt#20109, where I submitted a PR mostly to understand which tests had the subtests in question. I could have just searched by subtest name if it was possible.

@stephenmcgruer FYI in case you also find a recurring need for this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants