kprobe+eBPF tracing as an event source

[akindyakov]

Blueprint

To extend the type of information receiving from systems I'd like to propose the new way to get information about system events on Linux.

Existing mechanism

osquery already has event collection system for linux based on Linux Auditing System. It provides a lot of extremely useful information. However auditd has some well known limits by design.

Restrictions of auditd

The main constraint is a notable performance penalty. Audit information is received from the system via unix socket in text form. Which means kernel waste CPU resources to create those strings and filter them. osquery in it's turn has to parse them. Sounds ridiculous but in the scale of system events it has a big performance issue on high load systems.

Suggested solution

The possible substitution could be to use kprobes / kretprobes with eBPF to fetch intended data in kernel mode. With kprobe we could watch and trace almost any syscall with arguments, return values and some contexts like PID and TID of the caller.

Performance hit

I did the rough test to compare the system performance hit of considerable mechanisms: current osquery system event collection mechanism (aka auditd), kprobe watching every open syscall in the system. Both systems can trace the system call open, so I did a small binary which is open and close a file many times. Run it with auditd and with kprobe, also run it without anything to compare (etalon run).

parameter	etalon	kprobe	auditd
samples	13	13	14
average system time, s	1.54	2.71 (+76%)	6.23 (+304%)
min system time, s	1.46	2.59 (+77%)	6.06 (+315%)
variance system time, s	2.50	7.60	39.21

Performance hit for kprobe is notable lower than for auditd, thereby using kprobe can probably help us to reduce performance hit of system event recording mechanism.

Implementation plan

• Implement elementary wrapping eBPF structures such as maps, progs, output points based on perf events
• Implement kprobe managing code to connect it to eBPF program
• Implement on top of two previous parts event source for execve
• Create instruments to produce eBPF bytecode for any other system events

[akindyakov]

Implementation details of eBPF based event collection from linux kernel

The good new is the osquery lockdown is almost over, therefore I resume work on the new event source for the osquery on Linux. This is short term plan to have it done. Feel free to comment it or ask a questions. And also to comment coming PRs about topic.

Definitions:

• The arrows on the picture are to show data (events) flow.
• The "3d" boxes are for the osquery classes.
• And the ovals are for the linux kernel constructs which osquery use to retrieve event information.

On Linux since v4 we have direct access to the class of native tracing events in order to collect them in userspace. There is a set of Linux tracing event (see /sys/kernel/debug/tracing/events) which we can use to track syscalls, net connections and other things. To use them we should know the data format of the event (file format in event directory) and id number (file id). Also, the event should be enabled in kernel configuration and switched on by writing 1 to a file enabled. To make it easy to handle tracing events there is an osquery class tracing::NativeEvent.

To parse event and retrieve some context like PID, UID of the process caused the event eBPF program can be attached to the event by Linux performance monitoring mechanism (perf_event_open()). An osquery class was made to do it from cpp code: events::EbpfTracepoint. To load eBPF program the class ebpf::Program was made. The output of eBPF program could be received in userspace by Linux performance monitoring mechanism set up for every active CPU core (ebpf::PerfOutput). To find out to which perv channel program should write there is an eBPF map was made (ebpf::Map) to define a mapping from the CPU core number to file descriptor of the perf channel. Also to be able to read from the set of ebpf::PerfOutput file descriptors the ebpf::PerfOutputsPoll class was made.

For now, the main goal is to track syscalls from the kernel. We gonna start with kill (program code: events::genLinuxKillEnterProgram) and setuid (program code: events::genLinuxKillEnterProgram). Unfortunately the event of entering to a syscall and event of returning from it comes separately. They have to be joined. There are two main approaches to do it: in the kernel space by eBPF programs and in userspace. I chose the second one in order to keep eBPF programs small and lightweight and the number of eBPF structs less (in in-kernel approach one additional ebpf map is required for every syscall). To parse and receive "exit" events in user space there is an eBPF program: events::genLinuxExitProgram. It is exactly the same byte code for all exit events besides the code of the watching syscall. And of course, there is gonna be a matching class to quickly join exit and enter events to one syscall event.

This event is gonna be yet another event producer in new event framework from @fmanco.

[mike-myers-tob]

Related: #6571

[mike-myers-tob]

Closing because we have retired this approach in Remove unused/experimental ebpf code in favor of the ebpfpub library approach and Alessandro's BPF-based socket and process events tables (#6571) that were released in 4.6.0.