-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory leak with add_process_metadata and k8s manifest for Auditbeat #24890
Comments
Pinging @elastic/integrations (Team:Integrations) |
Pinging @elastic/security-external-integrations (Team:Security-External Integrations) |
Maybe it doesn't really matter but i'm using GKE with google COS as operating system on nodes. heap2.prof.zip - both processors active |
@mareckii thanks for the memory profiles, it actually looks like the problem is around the process cache in |
It seems that google COS is configured with a pid max of 2**22 (4194304), this is 128 times what I have in the machine where I tried (32768). If same memory usage ratio is maintained, a cache for so many pids would take more than 1.5GB. @mareckii could you confirm by checking the pid max in one of your affected machines? This can be checked with |
Hi,
if it helps memory dumps after few days: |
Thanks @mareckii. Yes, in this profile most of the memory in use is allocated by the process cache in This cache should have a different strategy for these cases. |
There seems to be a memory leak with
add_process_metadata
that is reproduced with the reference configuration provided to run Auditbeat in Kubernetes.This processor is used in this scenario to obtain the
container.id
from theprocess.id
, soadd_kubernetes_metadata
can enrich events. But issue is also reproduced whenadd_kubernetes_metadata
is not used.Tried to reproduce in a simpler scenario, only with docker, but memory usage of this processor didn't seem to increase beyond ~13MB. In the linked discuss issue there seems to be problems even with 1GB memory limits. Difference could be in the maximum number of pids allowed (
sysctl kernel.pid_max
).add_process_metadata
has a process cache whose entries are never cleaned, but the key is the pid, so its size is effectively limited by the maximum number of pids in the machine. The problem may be thatkernel.pid_max
can be quite big.Some stragegy should be applied to remove unneeded or expired entries from this cache.
For confirmed bugs, please report:
The text was updated successfully, but these errors were encountered: