-
Notifications
You must be signed in to change notification settings - Fork 897
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GKE - Falco Logs not appearing for up to 10 minutes #1291
Comments
Thanks for the report Can you share the following The exact helm chart and associated commandsRecently the community ported to this resource https://github.com/falcosecurity/charts/tree/master/falco#installing-the-chart Can you confirm you are using the must recent version here? How are you viewing the logs?Are you viewing the logs on all nodes in the cluster? Something like kubectl logs -l app=falco -f What is your burst set to?If you look here you can see the following: Note that 1000 seconds is between 16 and 17 minutes.
Probably the quickest way to debug this is to do send a quick batch of alerts consecutively and see if you start getting real-time alerts. When I demo I usually like to violate 10+ rules as quickly as possible. Usually, I do something like this: touch /usr/bin/1
touch /usr/bin/2
touch /usr/bin/3
cat /etc/shadow
touch /etc/1
touch /proc/1
cat ~/.bashrc
touch /root/1
touch /root/2
touch /root/3 Let me know of any of that doesn't help you out and I can spin up a GKE cluster tomorrow and tell you exactly what is going on. |
So I looked into the Helm chart (It looks like that is what you are using) and I think you are trying to pull logs on the same pod you are trying to trigger a rule with Which means I bet changing the values here or triggering more alerts could help get you closer to the state you are looking for. # The rate corresponds to one message every 30 seconds.
# A burst of 10 messages.
helm install falco --set ebpf.enabled=true,outputs.rate=.03333,outputs.maxRate=10 falcosecurity/falco |
seeing the same thing. when attempting to do a simple write to bin/ after using the helm values listed above: i also attempted: touch /usr/bin/1 |
It seems to me that this issue duplicates:
Could you confirm that's the same problem? Basically, the output log is delayed when Falco runs in a container in a non-interactive mode (ie. no TTY) and the activity is low (very few events per second). As soon as other events arrive, previous events appear immediately (so you won't notice this problem in a system that generates many events per second). I encountered the same problem many times, but AFAIK we don't have a fix yet. |
@danpopSD Investigated this with @leodido This happens because GKE distinguishes between a TTY and plain stdout while flushing logs. I achieved real time logs by doing this:
Nota beneRelated issues on kubernetes: kubernetes/kubernetes#57414 So at this point we need to make a decision here, we might want to enable tty in our manifests and helm chart. However, before doing that we need to investigate what are the boundaries and side effects of doing this. We need to answer some questions like:
|
Note that this issue is not GKE specific. I was able to reproduce it with docker (as explained in this comment) and others k8s installations as well. If we decided to enable TTY by default, we will need to update the docs regarding the docker usage too. |
this #1291 (comment) addressed my issue.. |
Since this issue pops out from time to time, I'd like to add the quick tl;dr here:
more information: https://falco.org/docs/alerts/#standard-output-buffering |
Describe the bug
GKE - Falco Logs not appearing for up to 10 minutes
How to reproduce it
Expected behaviour
logs/falco event notifications to be immediate or close to it vs 9-10 minutes..
Screenshots
Environment
Falco version: 0.23.0
Driver version: 96bd9bc560f67742738eb7255aeb4d03046b8045
{
"machine": "x86_64",
"nodename": "gke-popfalco1-default-pool-880376cf-qj8n",
"release": "4.19.109+",
"sysname": "Linux",
"version": "#1 SMP Mon Mar 16 06:27:01 PDT 2020"
}
GKE... Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.8-gke.15". (3 nodes)
Additional context
none
The text was updated successfully, but these errors were encountered: