falco inotify handler fails in k8s environment with ebpf driver

[dmitryroshchin]

Falco never apply rules files changes in our EKS environment when using ebpf driver. After adding any additional rules to falco_rules.yaml It just continues to work as if the file was never touched. From log messages below one can infer that inotify handler used to track those changes seems to be broken.

How to reproduce it

1. git clone https://github.com/falcosecurity/charts.git
2. cd charts/falco
3. change driver to ebpf in vaules.yaml
4. helm install falco ./ -n falco --values ./values.yaml
5. change falco_rules.yaml inside on of containers
6. notice the following error messages in falco pod logs:

Wed Oct 25 21:22:16 2023: SIGHUP received, restarting...
Syscall event drop monitoring:
   - event drop detected: 0 occurrences
   - num times actions taken: 0
21:21:57.227720194: Notice A shell was spawned in a container with an attached terminal (evt_type=execve user=root user_uid=0 user_loginuid=-1 process=sh proc_exepath=/bin/busybox parent=runc command=sh terminal=34816 exe_flags=EXE_WRITABLE container_id=c85a6d505d94 container_image=docker.io/falcosecurity/falcoctl container_image_tag=0.6.2 container_name=falcoctl-artifact-follow k8s_ns=falco k8s_pod_name=falco-7xqkv)
Events detected: 1
Rule counts by severity:
   NOTICE: 1
Triggered rules by rule name:
   Terminal shell in container: 1
Wed Oct 25 21:22:16 2023: Falco version: 0.36.1 (x86_64)

7. add new rule to falco_rules.yaml
8. falco ignores rules changes and never receives SIGHUP

Expected behaviour

At step 8 falco should reload and apply new rules

Environment

• Falco version:

Fri Oct 27 16:01:09 2023: Falco version: 0.36.1 (x86_64)
Fri Oct 27 16:01:09 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Falco version: 0.36.1
Libs version:  0.13.2
Plugin API:    3.1.0
Engine:        26
Driver:
  API version:    5.0.0
  Schema version: 2.0.0
  Default driver: 6.0.1+driver

• System info:

attached below

• Cloud provider or hardware configuration:
  AWS
• OS:
  Amazon linux
• Kernel:
  Linux falco-7xqkv 5.10.165-143.735.amzn2.x86_64 Pull falco chart from helm/charts #1 SMP
• Installation method:
  Kubernets, official helm chart

Additional context

Wed Oct 25 21:21:16 2023: Falco version: 0.36.1 (x86_64)
Wed Oct 25 21:21:16 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Wed Oct 25 21:21:16 2023: Loading rules from file /etc/falco/falco_rules.yaml
Wed Oct 25 21:21:16 2023: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Wed Oct 25 21:21:16 2023: Starting health webserver with threadiness 2, listening on port 8765
Wed Oct 25 21:21:16 2023: Loaded event sources: syscall
Wed Oct 25 21:21:16 2023: Enabled event sources: syscall
Wed Oct 25 21:21:16 2023: Opening 'syscall' source with BPF probe. BPF probe path: /root/.falco/falco-bpf.o
Wed Oct 25 21:22:16 2023: Falco version: 0.36.1 (x86_64)
Wed Oct 25 21:22:16 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Wed Oct 25 21:22:16 2023: Loading rules from file /etc/falco/falco_rules.yaml
Wed Oct 25 21:22:16 2023: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Wed Oct 25 21:22:16 2023: SIGHUP received, restarting...
Syscall event drop monitoring:
   - event drop detected: 0 occurrences
   - num times actions taken: 0
21:21:57.227720194: Notice A shell was spawned in a container with an attached terminal (evt_type=execve user=root user_uid=0 user_loginuid=-1 process=sh proc_exepath=/bin/busybox parent=runc command=sh terminal=34816 exe_flags=EXE_WRITABLE container_id=c85a6d505d94 container_image=docker.io/falcosecurity/falcoctl container_image_tag=0.6.2 container_name=falcoctl-artifact-follow k8s_ns=falco k8s_pod_name=falco-7xqkv)
Events detected: 1
Rule counts by severity:
   NOTICE: 1
Triggered rules by rule name:
   Terminal shell in container: 1
Wed Oct 25 21:22:16 2023: Falco version: 0.36.1 (x86_64)
Wed Oct 25 21:22:16 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Wed Oct 25 21:22:16 2023: Loading rules from file /etc/falco/falco_rules.yaml
Wed Oct 25 21:22:16 2023: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Wed Oct 25 21:22:16 2023: Starting health webserver with threadiness 2, listening on port 8765
Wed Oct 25 21:22:16 2023: Loaded event sources: syscall
Wed Oct 25 21:22:16 2023: Enabled event sources: syscall
Wed Oct 25 21:22:16 2023: Opening 'syscall' source with BPF probe. BPF probe path: /root/.falco/falco-bpf.o
Wed Oct 25 21:22:17 2023: Failed read with inotify handler, shutting down watcher...

[dmitryroshchin]

Here is the falco --support | jq .system_info output
falco_issue.txt

[Andreagit97]

ei @dmitryroshchin thank you for reporting! This seems the same issue reported here falcosecurity/libs#1448, it should be fixed in Falco 0.36.2, do you mind trying it? the helm chart to use should be the latest one (falco-3.8.4)

[Andreagit97]

/assign @Andreagit97

[dmitryroshchin]

Seems to be fixed in 0.36.2, many thanks!