Skip to content

Commit

Permalink
[SIEM][Auditbeat] Docs for system/socket dataset (elastic#13537)
Browse files Browse the repository at this point in the history
  • Loading branch information
adriansr authored Sep 9, 2019
1 parent 54860f7 commit b37e4ff
Show file tree
Hide file tree
Showing 2 changed files with 144 additions and 2 deletions.
144 changes: 143 additions & 1 deletion x-pack/auditbeat/module/system/socket/_meta/docs.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,147 @@

beta[]

This is the `socket` dataset of the system module.
This is the `socket` dataset of the system module. It allows to monitor network
traffic to and from running processes. It's main features are:

- Supports TCP and UDP sockets over IPv4 and IPv6.
- Outputs per-flow bytes and packets counters.
- Enriches the flows with https://www.elastic.co/guide/en/ecs/current/ecs-process.html[process]
and https://www.elastic.co/guide/en/ecs/current/ecs-user.html[user] information.
- Provides information similar to Packetbeat's flow monitoring with reduced CPU
and memory usage.
- Works on stock kernels without the need of custom modules, external libraries
or development headers.
This dataset does not analyze application-layer protocols nor provide any other
advanced features present in Packetbeat:
- Monitor network traffic whose destination is not a local process, as is the
case with traffic forwarding.
- Monitor layer 2 traffic, ICMP or raw sockets.

[float]
=== Implementation

It is implemented for Linux only and currently supports x86 (32 and 64 bit)
architectures.

The dataset uses
https://www.kernel.org/doc/Documentation/trace/kprobetrace.txt[KProbe-based event tracing]
to monitor TCP and UDP sockets over IPv4 and IPv6, providing flow monitoring
that includes byte and packet counters, as well as the local process and user
involved in the flow. It does so by plugin into the TCP/IP stack to generate
custom tracing events avoiding the need to copy network traffic to user space.

By not relying on periodic polling, this approach enables the dataset to perform
near real-time monitoring of the system without the risk of missing short lived
connections or processes.

[float]
=== Requirements

Features used by the `socket` dataset require a minimum Linux kernel version
of 3.12 (vanilla). However, some distributions have backported those features
to older kernels. The following (non-exhaustive) lists the different
distributions under which the dataset is known to work:

[options="header"]
|==============================================
| Distribution | kernel version | Works?
| CentOS 6.5 | 2.6.32-431.el6 | NO^<<anchor-1,[1]>>^
| CentOS 6.9 | 2.6.32-696.30.1.el6 | &#10003;
| CentOS 7.6 | 3.10.0-957.1.3.el7 | &#10003;
| Debian 8 | 3.16.0-6 | &#10003;
| Debian 9 | 4.9.0-8 | &#10003;
| Debian 10 | 4.19.0-5 | &#10003;
| SLES 12 | 4.4.73-5 | &#10003;
| Ubuntu 12.04 | 3.2.0-126 | NO^<<anchor-1,[1]>>^
| Ubuntu 14.04.6 | 3.13.0-170 | &#10003;
| Ubuntu 16.04.3 | 4.4.0-97 | &#10003;
| AWS Linux 2 | 4.14.138-114.102 | &#10003;
|==============================================

[[anchor-1]]
<<anchor-1,[1]>>: These systems lack
https://lore.kernel.org/patchwork/patch/399251/[PERF_EVENT_IOC_ID ioctl.]
Support might be added in a future release.

The dataset needs CAP_SYS_ADMIN and CAP_NET_ADMIN in order to work.

[float]
==== Kernel configuration

A kernel built with the following configuration options enabled is required:

- `CONFIG_KPROBE_EVENTS`: Enables the KProbes subsystem.
- `CONFIG_DEBUG_FS`: For kernels laking `tracefs` support (<4.1).
- `CONFIG_IPV6`.

These settings are enabled by default in most distributions.

The following configuration settings can prevent the dataset from starting:

- `/sys/kernel/debug/kprobes/enabled` must be 1.
- `/proc/sys/net/ipv6/conf/lo/disable_ipv6` must be 0
(IPv6 enabled in the loopback device).

[float]
==== Running on docker

The dataset can monitor the Docker host when running inside a container. However
it needs to run on a `privileged` container with `CAP_NET_ADMIN` and IPv6
support. The docker container running {beatname_uc} needs access to the host's
tracefs or debugfs directory. This is achieved by bind-mounting `/sys`.

[float]
=== Configuration

The following options are available for the `socket` dataset:

- `socket.tracefs_path` (default: none)

Must point to the mount-point of `tracefs` or the `tracing` directory inside
`debugfs`. If this option is not specified, {beatname_uc} will look for
the default locations: `/sys/kernel/tracing` and `/sys/kernel/debug/tracing`.
If not found, it will attempt to mount `tracefs` and `debugfs` at their
default locations.

- `socket.flow_inactive_timeout` (default: 30s)

Determines how long a flow has to be inactive to be considered closed.

- `socket.flow_termination_timeout` (default: 5s)

Determines how long to wait after a socket has been closed for out of order
packets. With TCP, some packets can be received shortly after a socket is
closed. If set too low, additional flows will be generated for those packets.

- `socket.perf_queue_size` (default: 4096)

The number of tracing samples that can be queued for processing. A larger value
uses more memory but reduces the chances of samples being lost when the system
is under heavy load.

- `socket.lost_queue_size` (default: 128)

The number of lost samples notifications that can be queued.

- `socket.ring_size_exponent` (default: 7)

Controls the number of memory pages allocated for the per-CPU ring-buffer
used to receive samples from the kernel. The actual amount of memory used is
Number_of_CPUs x Page_Size(4KB) x 2^ring_size_exponent^. That is 0.5 MiB of RAM
per CPU with the default value.

- `socket.clock_max_drift` (default: 100ms)

Defines the maximum difference between the kernel internal clock and
the reference time used to timestamp events.

- `socket.clock_sync_period` (default: 10s)

Controls how often clock synchronization events are generated to measure drift
between the kernel clock and the dataset's reference clock.

- `socket.guess_timeout` (default: 15s)

The maximum time an individual guess is allowed to run.
2 changes: 1 addition & 1 deletion x-pack/auditbeat/module/system/socket/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ type Config struct {
LostQueueSize int `config:"socket.lost_queue_size,min=1"`

// ErrQueueSize defines the size of the error queue. A single error is fatal.
ErrQueueSize int `config:"socket.err_buffer_size,min=1"`
ErrQueueSize int `config:"socket.err_queue_size,min=1"`

// RingSizeExp configures the exponent size for the per-cpu ring buffer used
// by the kernel to pass tracing events.
Expand Down

0 comments on commit b37e4ff

Please sign in to comment.