A service has been introduced which runs a dockerised image of journald-cloudwatch-logs. This service forwards journald logs to AWS CloudWatch to a LogGroup with the name of .ClusterName and is run on all nodes (Etcds, Controllers and Workers).
journald-cloudwatch-logs is a goLang project https://github.com/saymedia/journald-cloudwatch-logs.
The default docker image jollinshead/journald-cloudwatch-logs is a wrapper around the go binary of journald-cloudwatch-logs.
This feature is disabled by default and configurable in cluster.yaml:
cloudWatchLogging:
enabled: false
retentionInDays: 7
The docker image is also configurable:
journaldCloudWatchLogsImage:
repo: "jollinshead/journald-cloudwatch-logs"
tag: "0.1"
rktPullDocker: true
During kube-aws up/update, filtered Journald logs can be printed to stdout. This may assist debugging. The format of the messages are:
TimePassed NodeName: "LogMessage"
For example:
+00:04:51 ip-10-29-29-100.us-west-2.compute.internal: "check-certification-validity.service: Failed to run 'start-pre' task: No such file or directory"
+00:04:52 ip-10-29-29-100.us-west-2.compute.internal: "check-certification-validity.service: Failed with result 'resources'."
+00:04:53 ip-10-29-29-100.us-west-2.compute.internal: "kubelet.service: Failed with result 'exit-code'."
This feature is configurable in cluster.yaml under the cloudWatchLogging section, and requires cloudWatchLogging to be enabled. ( Default values: )
cloudWatchLogging:
enabled: false
imageWithTag: jollinshead/journald-cloudwatch-logs:0.1
retentionInDays: 7
localStreaming:
enabled: true
filter: `{ $.priority = "CRIT" || $.priority = "WARNING" && $.transport = "journal" && $.systemdUnit = "init.scope" }`
interval: 60
NOTE: Due to high initial entropy, .service failures may occur during the early stages of booting. In this context Entropy refers to the disorder of .services (starting, failing, restarting).
By default the filter is configured for .service failures and messages flagged as 'critical'. See the official AWS documentation for more information.
Since some messages are produced frequently, to avoid excessive spam, an 'interval' parameter is provided. This 'interval' value determines the time between printing two identical messages to stdout.