Improve support of Beats on Kubernetes restricted environments

[jsoriano]

In general, monitoring systems with Beats requires certain privileges, but they can work with limited functionality without them.

This issue keeps track of known issues running Beats on Kubernetes restricted environments, to explore alternatives that would allow to use more functionality with less privileges. Some of the issues will probably apply to other environments too.

Restricted environments have some limitations, for example the restricted Security Context Constraints in Openshift include:

• Privileged mode not allowed.
• Some capabilities are forbidden.
• Limited set of volumes allowed (no hostPath).
• Host namespaces not allowed.
• User ID must be in a range of ids defined in the project (so the uid will be unknown beforehand, and runAsUser: 0 is not allowed).

Known issues

Data persistence

Data persistence is required in beats in some features:

• meta.json (with stored UUID, used in stack monitoring?)
• Filebeat registry
• Auditbeat datastores (not always required but recommended)
• [TBD] Metadata on-disk caches

Beats reference manifests currently rely on being able to create a directory in the host with hostPath. This is not possible in restricted environments because hostPath is not allowed, and because directories created by hostPath can only be written by root (uid 0 on host).

Possible improvements/alternatives:

• Document alternatives with persistent volume claims (will depend on available volume providers)
• Explore deployment as StatefulSet on cases where DaemonSet is not needed.
• Depend on the local storage operator.
• Use an external database to store data instead of files.

Auditing

Auditing with auditbeat requires to be run with audit capabilities, on host pid namespace and with uid 0.

There is probably no possible alternative, but we could do some improvements:

• Explicitly document these requirements somewhere.
• Offer alternative configurations for Auditbeat to use other features that don't require so much privileges.

Host metadata

Retrieving host metadata requires to run beats on host namespace, this is not allowed on restricted environments.

Possible alternatives/improvements:

• Document this limitation in the manifests, so affected parts can be commented out, or they are commented out by default.
• Use information obtained from the downward API (spec.nodeName as the host name, though not always the same, status.hostIP as the host IP).
• Look for alternative ways of collecting host information using existing APIs.

Host network monitoring

Monitoring the host network relies on running Beats on the host network namespace.

Possible alternatives/improvements:

• Add comments about this in the reference manifests/docs.
• Comment out by default features that require running on host namespaces.
• Try to collect more information from host filesystems like /proc.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!