gaby: monitor timed.Watcher progress

[rsc]

Known timed.Watchers should advance their "latest" time in the database.
We should have a health check endpoint that reports whether the known
watchers are too old. This will help diagnose cron-based jobs getting stuck,
like when GitHub got stuck due to the non-issue issue event.

[jba]

I set up a metric for watcher latest times.

To be done: an alert based on the metric.

[hyangah]

The current metric exports DBTime which looks like a logical time.
What will be the reasonable alerting condition? (staying on the same value for N hours in real clock?)

[jba]

Not making progress is certainly a necessary condition. But we don't know if that's because something's stuck, or because there's nothing new. With the Go issue tracker, no progress in an hour would almost certainly be a problem. But with discussions, for example, there might be no progress for days. So it's tricky.