[DISCUSS] Support Linux Monitoring(by ebpf-tools)

[yswdqz]

Motivation

eBPF is a revolutionary technology with origins in the Linux kernel that can run sandboxed programs in a privileged context such as the operating system kernel. It is used to safely and efficiently extend the capabilities of the kernel without requiring to change kernel source code or load kernel modules.

In recent years, eBPF has played an increasingly significant role in the field of observability. The Apache SkyWalking community also has its own eBPF tool — SkyWalking Rover. However, Rover is not suitable for simple eBPF performance probing (such as basic I/O status monitoring). BCC / bpftrace provide many ebpf tools to help us perform these simple runtime status checks:

Therefore, we can leverage these tools to build a dedicated eBPF performance monitoring dashboard. We can implement basic performance monitoring using just a single dashboard, similar to Brendan Gregg's "Linux Performance Analysis in 60 seconds".

Architecture Graph

Let's start by introducing one component: ebpf_exporter
ebpf_exporter is a Prometheus exporter for custom eBPF metrics and OpenTelemetry traces. Motivation of this exporter is to allow us to write eBPF code and export metrics that are not otherwise accessible from the Linux kernel. With this component, we can easily convert the output of eBPF command-line programs into the OpenTelemetry format.

Proposed Changes

The first set of performance metrics I aim to contribute will come from the following tools, They will complement the existing vm-monitor.

BPF Program Name	Desctription	Metric Type	Metric Name
accept-latency	Latency histogram for how long sockets sit in accept queue	histogram	ebpf_exporter_accept_latency
biolatency	Block IO latency histogram	histogram	ebpf_exporter_bio_latency_seconds
ext4dist	ext4 IO latency histogram	histogram	ebpf_exporter_ext4_latency_seconds
cachestat	Page cache operation counters by type	counter	ebpf_exporter_page_cache_ops_total
softirq-latency	Latency histogram for how long it takes from softirq raise to entry	histogram	ebpf_exporter_softirq_entry_latency_seconds
oomkill	Count global and cgroup level OOMs	counter	ebpf_exporter_oom_kills_total
tcp_retransmit	Total count of IPv4 TCP retransmissions	counter	ebpf_exporter_tcp_retransmit_ipv4_packets_total
kfree_skb	Number of calls into skb:kfree_skb with a per-reason breakdown	counter	ebpf_exporter_kfree_skb_total
cfs-throttling	Latency histogram for CFS throttling	histogram	ebpf_exporter_cfs_throttling_seconds
unix-socket-backlog	Unix socket backlog size	histogram	ebpf_exporter_unix_socket_backlog
etc

Here are some things I hope to contribute in the near future：

1. bio-trace
2. exec-trace
3. sched-trace
4. sock-trace
5. cfs-throttling-trace
   These programs can provide trace data for us to observe system performance in greater detail.

There are also some things I will contribute in the later future:

1. execsnoop
2. opensnoop
3. tcpconnect
4. tcpaccept
5. gethostlatency
6. memleak
7. etc
   These are very important programs, but they are not directly supported by ebpf_exporter and require manually writing configuration files.

Imported Dependencies libs and their licenses.

No new dependency.

Compatibility

No breaking changes.

[wu-sheng]

ebpf_exporter is a Prometheus exporter for custom eBPF metrics and OpenTelemetry traces.

So, you are proposing to accept metrics only through this SWIP. And do you mean this is specific for VM monitoring extension? Further, what do you suggest visualizing?

[yswdqz]

Yes, I was initially thinking that using another layer might be better, but on second thought, maybe sharing one dashboard is a good idea. This way, the two sets of data can collaborate better with each other.

However, it seems that the UI doesn't currently support nested tabs? I tried adding two tabs in a dashboard of the online demo, and the page threw errors.

[wu-sheng]

I am not sure. But why a nested tab? This is an uncommon case.

[yswdqz]

[wu-sheng]

Why CPU needs a separate tab? Isn't that a widget only?

[yswdqz]

Why CPU needs a separate tab? Isn't that a widget only?

It depends on how many metrics we will use（for now it is not many, widgets is good）. If there are too many, the logic might not be very clear by a widget or widgets(perhaps calling it 'cpu' isn't quite accurate, and the correct term might be 'scheduler')

[wu-sheng]

fyi @mrproliu When you are back from the vacation, I would like to hear your thoughts about this.