Health/SLA checks

[foolip]

Moving this from web-platform-tests/results-collection#423.

We will have certain goals for run cadence and PR-to-results latency, captured in Ecosystem Infra 2018 OKRs as "wpt.fyi results are available within 1 hour" and issue Test every commit of web-platform-tests within 1 hour.

Whatever monitoring is in place for the runners, we should also have our own checks, especially to notice if we're getting fewer hash-aligned runs than we think we need/should.

[foolip]

I've now added a section to https://foolip.github.io/ecosystem-infra-rotation/ for stable and experimental runs. I picked daily as the SLA for stable, and weekly for experimental. I'd like experimental to be daily as well, but that's blocked on faster Edge and Safari runs.

@lukebjerring is there any additional monitoring you'd like to see?

[lukebjerring]

Nice. No, this should be enough. Though, when a run fails to upload it'd be good to know proactively rather than wait for the cop to notice it.
That's an issue for the running infra side, though.

[foolip]

@Hexcles, if runs are submitted to the results receiver but never make it out the other end, is that something that shows up as a problem in the GAE dashboard or email?

[Hexcles]

@foolip yes. There'll be Stackdriver error reports on GCP dashboard. And if you subscribe to notification (which I did), you'll get emails.

[foolip]

OK, let's resolve this!