Add github checks output support to stability check #24741
Conversation
stephenmcgruer
changed the title
stephenmcgruer
force-pushed
the
stability-output-md
branch
from
e39359f
to
0f5820f
Compare
stephenmcgruer
changed the title
wpt-pr-bot
added
ci
dom
infra
wptrunner
wpt-pr-bot
requested review from
annevk,
Hexcles,
jdm,
jgraham,
LukeZielinski and
zqzhang
stephenmcgruer
force-pushed
the
stability-output-md
branch
from
0f5820f
to
aa50e16
Compare
Due to taskcluster/taskcluster#3191, this requires us to set both textArtifactName and annotationsArtifactName. After that fix is rolled out to Taskcluster, we can switch to just using textArtifactName.
stephenmcgruer
force-pushed
the
stability-output-md
branch
from
aa50e16
to
6be633b
Compare
stephenmcgruer
force-pushed
the
stability-output-md
branch
from
6be633b
to
b42847e
Compare
stephenmcgruer
force-pushed
the
stability-output-md
branch
from
48d5d82
to
38d3cc0
Compare
There was a problem hiding this comment.
Please take another look. I don't love my code here...
On a more meta level, I'm also having trouble in framing the instructions for test authors. Specifically, I want to be able to say something like "please use the printed WPT command to reproduce", but the command as dumped will have multiple flags they probably don't want to use (i.e. --github-checks-text-file=/home/test/artifacts/checkrun.md and --affected base_head). So far I've changed it to a weaker sentence, but maybe I should be even more explicit, something like:
Please try to reproduce; see the above WPT command for what the bot ran, but note you may need to remove some flags to run locally.
Which is still confusing to test authors :/.
There was a problem hiding this comment.
This is pretty exciting!
I don't have a great suggestion for showing the command that you should run to get things working locally without some terrible hack where you have a blocklist of flags to remove. We never quite managed to solve this issue for gecko ci (although in that case things are worse because the ci uses a totally different entrypoint compared to the local runs).
stephenmcgruer
force-pushed
the
stability-output-md
branch
from
38d3cc0
to
e4fab44
Compare
stephenmcgruer
force-pushed
the
stability-output-md
branch
from
e4fab44
to
6939731
Compare
stephenmcgruer
force-pushed
the
stability-output-md
branch
from
6939731
to
194de61
Compare
|
Reverted the test change, will land this once CI is green. For posterity, this is what it looks like:
|
No description provided.