Configurable frequency of health check output updates

[armon]

Currently, if a health check's output changes, the new output is immediately written to the catalog. If the check does not change state (stays passing), but has output that varies (includes timestamp, variable output, etc), then this causes lots of useless writes.

This reduces scalability and causes the blocked queries to return rapidly with idempotent writes. Instead, if a check does not change state, we should be able to reduce the frequency of updates. This means the health output will be relatively up-to-date for a stable check, and immediately up-to-date for a recently changed check. The upside is much reduced writes.

[andrewwatson]

oh, is that what the output field in the UI is? I hadn't made that connection yet! 😄

[armon]

Yeah, the Output captures the stdout/stderr of a health check, while Notes are just human readable description an operator inputs.