Time the runtime of Drain and DrainOnShutdown #578
Conversation
| @@ -754,6 +757,8 @@ public void runMayThrow() throws InterruptedException | |||
| ScheduledExecutors.nonPeriodicTasks.shutdown(); | |||
| if (!ScheduledExecutors.nonPeriodicTasks.awaitTermination(1, MINUTES)) | |||
| logger.warn("Miscellaneous task executor still busy after one minute; proceeding with shutdown"); | |||
|
|
|||
| logger.info("DrainOnShutdown completed in {} ms", SafeArg.of("ms", watch.elapsed(TimeUnit.MILLISECONDS))); | |||
There was a problem hiding this comment.
Would it not be simpler to just add metrics? Or is there any advantages to using the logs?
There was a problem hiding this comment.
Since we poll the metrics endpoints, it's likely that a metric reported just before shutdown would be missed.
LogQL supports querying logs as if they were metrics, so there is no benefit to using metrics here over a log.
Metrics are also a bit more complicated to set up, and are better suited for values that should be continuously sampled, instead of just an infrequently fired event.
There was a problem hiding this comment.
Can you add logs at the start of each function (so we understand if a SIGKILL was sent before the thread completed) flush each of the logs?
There was a problem hiding this comment.
added a log to drainOnShutdown
the setMode calls i.e:setMode(Mode.DRAINING, "starting drain process", true); will emit a log
void setMode(Mode m, @Safe String msg, boolean log)
{
operationMode = m;
if (log)
logger.info(m.toString(), SafeArg.of("msg", msg));
else
logger.debug(m.toString(), SafeArg.of("msg", msg));
}
These runtimes will be useful to determine reasonable settings for K8s termination grace period.
LogQL can be used to treat these logs lines as metrics for aggregating these values across many clusters.