klib(radar): enhance - /shutdowns report (as we do for /boots), /machine-stats cpu times (as we do for memory), /crashes report also when kernel crashed

[rinor]

• /shutdowns - endpoint/functionality to report kernel/program stop/shutdown with enough information to identify
  • shutdown reason - i.e: normal instance shutdown/power-off requested, program terminated normal/failure, identifiable other failure that can still be recorded.
  • shutdown program status - it might be the program exit_code when available, or any other predefined/identifiable code (maybe negative) that report that the program did not terminate in time/or on it's own.

It might look smth like the following (just a general idea, since have yet to investigate what can be reported and what not)

{
  "bootID": 1234,
  "reason": 0(shutdown)|1(failure)|other_codes_if_identifiable
  "status": exit_code u(0 - 255)|- killed (i.e: because of UNIX_SHUTDOWN_TIMEOUT_SECS)
}

• /machine-stats - cpu usage - if feasible without introducing locks/delays in kernel, report all times necessary to calculate "instance" cpu "usage".

• /crashes - identify kernel crash - when the kernel crashes and nothing is sent/executed in shutdown handlers, there might be no "dump" to be sent upon (re)start, so we just get the /boots notification, but no crashes.
  Now this might not be needed assuming that we can treat the lack of /shutdowns notification as a severe kernel crash, however the naive idea would be for the radar klib to set "smth" upon boot, that the kernel "has to clean" upon "correct" shutdown. It might be an additional carved space or the same we use for KLOG_DUMP. Upon start, report it as crash and (re)set that "smth" so that kernel can identify that it's not a program crash_dump and clean it.

Those are just some initial raw ideas, for real needs, that could help.

[eyberg]

the radar klib is connected to a hosted service https://nanovms.com/radar much in the same way cloudwatch is connected to aws' cloudwatch; as such there are other server-side components to this so while some of this might get incorporated in the future that roadmap is entirely driven by the radar svc roadmap itself and we'd only make changes for that particular klib in conjunction w/changes for radar