Retrying to submit active-selection-045.html and a support file

[TalbotG]

active-selection-045.html (with a font size of 80px)

support/100x100-red.png

This is the bottom half of
#24714

[wpt-pr-bot]

There are no reviewers for this pull request. Please reach out on W3C's irc server (irc.w3.org, port 6665) on channel #testing (web client) to get help with this. Thank you!

[TalbotG]

Dear fellow advanced GitHub colleagues @web-platform-tests/admins ,

Please help me understand why my test active-selection-045.html fails and fails with integration check wpt-chrome-dev-stability . I have been trying, re-trying and re-re-trying persistently to do so in
#24723
and in here
#24726

I have lots of questions right now ... but let's start with why active-selection-045.html is too much for wpt-chrome-dev-stability CI.

Thank you,

Gérard

[jgraham]

Test	Subtest	Results	Messages
/css/css-pseudo/active-selection-045.html		FAIL: 2/10, PASS: 8/10

[jgraham]

I strongly suspect this should use reftest-wait to ensure the script runs before the screenshot is taken.

[stephenmcgruer]

I strongly suspect this should use reftest-wait to ensure the script runs before the screenshot is taken.

Agreed. Loading the test in a browser, there is a flicker of a frame drawn with the red-line existing before it disappears. @TalbotG , https://web-platform-tests.org/writing-tests/reftests.html#controlling-when-comparison-occurs documents the reftest-wait class, and you can look at a test like css/css-pseudo/marker-content-013.html for an example of it being used.

[TalbotG]

@jgraham , @stephenmcgruer ,

I have added the reftest-wait class to the root element in active-selection-045.html test in
#24714
[Edit: I have added a script instruction to remove class reftest-wait after script run a few min ago. Thank you jgraham.]
I have never use reftest-wait before and never thought this would be necessary here. But it makes sense, furthermore and especially if you see "a flicker of a frame drawn with the red-line existing before it disappears" ... which I do not.

What bugs me the most is that when a integration check reports a failure, there is no way for me to read some kind of error message describing somehow what failed or what was wrong or giving me some clue. So, I can not develop an autonomy, an independence with regards to GitHub and Continuous Integration checks. Just look:
#24710
#24714
#24715
#24723
#24726
are all about or related to active-selection-045.html test failure ... and even @fantasai could not figure out either what was wrong with 045 test.

[TalbotG]

Let's say I submit 10 tests in a PR. An integration checks fails. Ideally, an error message somewhere would inform about which test(s) has/have caused the failure. If, say, 2 tests have caused 1 CI check to fail, then I would still be able to re-submit a PR with the 8 "successful" tests. The way it is right now, I am not informed of which test(s) fail(s) and no clue given as to why it/they fail.

[stephenmcgruer]

What bugs me the most is that when a integration check reports a failure, there is no way for me to read some kind of error message describing somehow what failed or what was wrong or giving me some clue. So, I can not develop an autonomy, an independence with regards to GitHub and Continuous Integration checks. Just look:

I think there are two levels to this. Firstly, there is accessing the details of the check, and secondly there is understanding what the failure means.

For the first, the details are available today, but we do also have long-term plans to make them more easily accessible. Today, it requires clicking through three levels: 'details' on GitHub, then 'View task in Taskcluster', then 'View Live Log':

In the future, we intend to extract information from the logs and put it in the first GitHub Checks output object (the area where today it just says "Verify that all tests affected by a pull request are stable when executed in chrome.").

The second part of the question is: how do you understand from the checks what has gone wrong. This is a much harder problem, and it's not clear to me how far along that road we should go. Today, the logs will tell you (a) what the inconsistency was (**FAIL: 2/10, PASS: 8/10** for your test), and (b) it will above have logged the result for each test along the way. I note that for reftests we don't seem to log the base64-encoded png of the screenshots, perhaps we should do that (or perhaps we should store the pngs in the artifacts space, though I'm not sure how much overall space we have available there). Going much further than this seems like a huge investment in guessing what the test may be doing wrong - there are so, so many ways to cause flakes :)

[TalbotG]

The details are available (...) through 'details' on GitHub, then 'View task in Taskcluster', then 'View Live Log': just finding this was not obvious to me. And once in View Live Log, I could not scroll that page with the vertical scrollbar. I only could roll the mousewheel to get into the file... once I noticed that I could scroll that 'View Live Log' page with the mousewheel.

The re-run link in the Details page: it did not work... or I could not make it work ... or it was re-running but I could not be sure of it (no visual feedback).

etc

[TalbotG]

I must close this PR and delete this branch now that
#24714
has merged.

[stephenmcgruer]

No disagreement to any of that; that is exactly what #15412 covers. We're aware that this is painful, and we're sorry and we'd like to fix it, but we have limited bandwidth. Slowly making progress in that direction - the big hurdle of moving to Taskcluster Checks is done and now we 'just' need to rewrite part of the stability checker to output formatted data that GitHub can read.

[TalbotG]

Stephen,

If/when I see a vertical scrollbar in a webpage, then I should (and I expect) to be able to scroll it with [PgDn]/[PgUp] or navigation up/down arrow keys. There is a counter-intuitive, disorienting visual feedback in that View Live Log page.

If a particular CI check fails, then I wish (and wished) I would be informed somehow/somewhere about which test(s) in my PR is causing issues or problems from any of the 'details' or 'View task in Taskcluster' or 'View Live Log' "hoops" (sorry if my english is awkward here).

How to convey a descriptive error message to the submitter is indeed a huge complex task. I have unfortunately no constructive suggestion or idea on that specific issue.

[stephenmcgruer]

If/when I see a vertical scrollbar in a webpage, then I should (and I expect) to be able to scroll it with [PgDn]/[PgUp] or navigation up/down arrow keys. There is a counter-intuitive, disorienting visual feedback in that View Live Log page.

Yeah, the Taskcluster UI is... a little funky. I have also found trouble getting it to scroll. Feel free to open an issue https://github.com/taskcluster/taskcluster/issues/

If a particular CI check fails, then I wish (and wished) I would be informed somehow/somewhere about which test(s) in my PR is causing issues or problems from any of the 'details' or 'View task in Taskcluster' or 'View Live Log' "hoops" (sorry if my english is awkward here).

Yep, no disagreement. Long term I'm aiming for 1 hop ('details') and that's it. I personally would also like a bot that comments on the PR with details, but that's contentious around here (a lot of folks object to bot comments).

[TalbotG]

the logs will tell you (a) what the inconsistency was (FAIL: 2/10, PASS: 8/10 for your test)

I did not see this. I still today (and yesterday and 2 days ago) do not see this. I do not know where you see/saw this particular information ! I do see "2 failing and 17 successful checks".

[stephenmcgruer]

At the bottom of the logs for wpt-chrome-dev-stability: