Custom checks showing up on all commits on master, comparing against master

[foolip]

In https://github.com/web-platform-tests/wpt/commits/master, every commit gets wpt.fyi "chrome[experimental] and "firefox[experimental]" checks. Example:
https://github.com/web-platform-tests/wpt/runs/38327762
https://github.com/web-platform-tests/wpt/runs/38327768

This could be very useful if the comparison was against the previous commit on master.

[foolip]

A comparison against the previous run of the same product would in particular come in handy after landing a change like web-platform-tests/wpt#14309 where some regressions might have slipped through.

[lukebjerring]

I think the checks you're pointing at are (incorrectly) comparing the same run against itself, since the "latest master run" is the run itself. I'll use this issue to follow up on ensuring that we find the previous run in those cases.

As for making it show up for everyone, you'll just need to enable checksAllUsers. This isn't added in the /admin/flags view just yet - will add it.

[Hexcles]

By the way, the user whitelist seems to be not working. All commits from all users on master have the wpt.fyi custom checks.

[lukebjerring]

It was that way a couple days ago, so you might just be seeing leftovers from that bug, but if you scroll the list a few don't have the check (e.g. commits from youenff)

[foolip]

For commit 226f3b5 3 checks are showing up:

But for the preceding commit no checks. Not sure why :)

[lukebjerring]

The pr author is whitelisted, so their merge triggers a check suite

[gsnedders]

The pr author is whitelisted, so their merge triggers a check suite

Why does authorship matter when the commit is on the wpt repo itself? If a commit has been pushed to the WPT repo itself, it's been pushed by someone we should consider trustworthy.

[foolip]

The list was just to try it out with a small number of people, not for security reasons.

[lukebjerring]

Yah, temporary measure to reduce noise during unstable development.

[lukebjerring]

This was fixed by #872