Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[journald] Add 8.x support #4037

Merged
merged 1 commit into from
Aug 18, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/journald/_dev/build/build.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
dependencies:
ecs:
reference: git@1.11
reference: git@v8.4.0-rc1
4 changes: 3 additions & 1 deletion packages/journald/_dev/build/docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,6 @@ The journald input reads the log data and the metadata associated with it.

The journald input is available on Linux systems with `systemd` installed.

{{fields "log"}}
{{event "log"}}

{{fields "log"}}
8 changes: 8 additions & 0 deletions packages/journald/changelog.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,12 @@
# newer versions go on top
- version: "0.0.4"
changes:
- description: Update Kibana requirement to support 8.x.
type: enhancement
link: https://github.com/elastic/integrations/pull/4037
- description: Map `message_id` to ECS `event.code`.
type: enhancement
link: https://github.com/elastic/integrations/pull/4037
- version: "0.0.3"
changes:
- description: Add documentation for multi-fields
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,11 @@ processors:
target_field: log.syslog.identifier
ignore_missing: true

- rename:
field: message_id
target_field: event.code
ignore_missing: true

# Cleanup an empty syslog object.
- remove:
if: ctx?.syslog != null && ctx.syslog instanceof Map && ctx.syslog.isEmpty()
Expand Down
2 changes: 2 additions & 0 deletions packages/journald/data_stream/log/fields/ecs.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
- name: ecs.version
external: ecs
- name: event.code
external: ecs
- name: host.hostname
external: ecs
- name: host.id
Expand Down
102 changes: 102 additions & 0 deletions packages/journald/data_stream/log/sample_event.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
{
"@timestamp": "2020-07-22T13:17:10.012Z",
"agent": {
"ephemeral_id": "27e2a00a-dab2-4790-8d45-29ad272d0392",
"id": "bef8099b-68f6-4621-8089-2229b35a669d",
"name": "docker-fleet-agent",
"type": "filebeat",
"version": "8.3.2"
},
"data_stream": {
"dataset": "journald.log",
"namespace": "ep",
"type": "logs"
},
"ecs": {
"version": "8.0.0"
},
"elastic_agent": {
"id": "bef8099b-68f6-4621-8089-2229b35a669d",
"snapshot": false,
"version": "8.3.2"
},
"event": {
"agent_id_status": "verified",
"code": "ec387f577b844b8fa948f33cad9a75e6",
"created": "2022-08-18T18:14:11.588Z",
"dataset": "journald.log",
"ingested": "2022-08-18T18:14:15Z",
"kind": "event"
},
"host": {
"hostname": "sleipnir",
"id": "505afdafda3b4f33a63749ae39284742"
},
"input": {
"type": "journald"
},
"journald": {
"custom": {
"available": "0",
"available_pretty": "0B",
"current_use": "1023455232",
"current_use_pretty": "976.0M",
"disk_available": "6866636800",
"disk_available_pretty": "6.3G",
"disk_keep_free": "1466253312",
"disk_keep_free_pretty": "1.3G",
"journal_name": "System journal",
"journal_path": "/var/log/journal/505afdafda3b4f33a63749ae39284742",
"limit": "977502208",
"limit_pretty": "932.2M",
"max_use": "977502208",
"max_use_pretty": "932.2M"
},
"gid": 0,
"host": {
"boot_id": "fa3c2e3080dc4cd5be5cb5a43e140d51"
},
"pid": 19317,
"process": {
"capabilities": "25402800cf",
"command_line": "/lib/systemd/systemd-journald",
"executable": "/lib/systemd/systemd-journald",
"name": "systemd-journal"
},
"uid": 0
},
"log": {
"syslog": {
"facility": {
"code": 3
},
"identifier": "systemd-journald",
"priority": 6
}
},
"message": "System journal (/var/log/journal/505afdafda3b4f33a63749ae39284742) is 976.0M, max 932.2M, 0B free.",
"process": {
"args": [
"/lib/systemd/systemd-journald"
],
"args_count": 1,
"command_line": "/lib/systemd/systemd-journald",
"pid": 19317
},
"systemd": {
"cgroup": "/system.slice/systemd-journald.service",
"invocation_id": "7c11cda63635437bafe21c92851618a8",
"slice": "system.slice",
"transport": "driver",
"unit": "systemd-journald.service"
},
"tags": [
"forwarded"
],
"user": {
"group": {
"id": "0"
},
"id": "0"
}
}
115 changes: 111 additions & 4 deletions packages/journald/docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,113 @@ The journald input reads the log data and the metadata associated with it.

The journald input is available on Linux systems with `systemd` installed.

An example event for `log` looks as following:

```json
{
"@timestamp": "2020-07-22T13:17:10.012Z",
"agent": {
"ephemeral_id": "27e2a00a-dab2-4790-8d45-29ad272d0392",
"id": "bef8099b-68f6-4621-8089-2229b35a669d",
"name": "docker-fleet-agent",
"type": "filebeat",
"version": "8.3.2"
},
"data_stream": {
"dataset": "journald.log",
"namespace": "ep",
"type": "logs"
},
"ecs": {
"version": "8.0.0"
},
"elastic_agent": {
"id": "bef8099b-68f6-4621-8089-2229b35a669d",
"snapshot": false,
"version": "8.3.2"
},
"event": {
"agent_id_status": "verified",
"code": "ec387f577b844b8fa948f33cad9a75e6",
"created": "2022-08-18T18:14:11.588Z",
"dataset": "journald.log",
"ingested": "2022-08-18T18:14:15Z",
"kind": "event"
},
"host": {
"hostname": "sleipnir",
"id": "505afdafda3b4f33a63749ae39284742"
},
"input": {
"type": "journald"
},
"journald": {
"custom": {
"available": "0",
"available_pretty": "0B",
"current_use": "1023455232",
"current_use_pretty": "976.0M",
"disk_available": "6866636800",
"disk_available_pretty": "6.3G",
"disk_keep_free": "1466253312",
"disk_keep_free_pretty": "1.3G",
"journal_name": "System journal",
"journal_path": "/var/log/journal/505afdafda3b4f33a63749ae39284742",
"limit": "977502208",
"limit_pretty": "932.2M",
"max_use": "977502208",
"max_use_pretty": "932.2M"
},
"gid": 0,
"host": {
"boot_id": "fa3c2e3080dc4cd5be5cb5a43e140d51"
},
"pid": 19317,
"process": {
"capabilities": "25402800cf",
"command_line": "/lib/systemd/systemd-journald",
"executable": "/lib/systemd/systemd-journald",
"name": "systemd-journal"
},
"uid": 0
},
"log": {
"syslog": {
"facility": {
"code": 3
},
"identifier": "systemd-journald",
"priority": 6
}
},
"message": "System journal (/var/log/journal/505afdafda3b4f33a63749ae39284742) is 976.0M, max 932.2M, 0B free.",
"process": {
"args": [
"/lib/systemd/systemd-journald"
],
"args_count": 1,
"command_line": "/lib/systemd/systemd-journald",
"pid": 19317
},
"systemd": {
"cgroup": "/system.slice/systemd-journald.service",
"invocation_id": "7c11cda63635437bafe21c92851618a8",
"slice": "system.slice",
"transport": "driver",
"unit": "systemd-journald.service"
},
"tags": [
"forwarded"
],
"user": {
"group": {
"id": "0"
},
"id": "0"
}
}
```

**Exported fields**

| Field | Description | Type |
Expand All @@ -16,6 +123,7 @@ The journald input is available on Linux systems with `systemd` installed.
| data_stream.namespace | Data stream namespace. | constant_keyword |
| data_stream.type | Data stream type. | constant_keyword |
| ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword |
| event.code | Identification code for this event, if one exists. Some event sources use event codes to identify messages unambiguously, regardless of message language or wording adjustments over time. An example of this is the Windows Event ID. | keyword |
| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword |
| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword |
| input.type | | keyword |
Expand Down Expand Up @@ -56,11 +164,11 @@ The journald input is available on Linux systems with `systemd` installed.
| log.syslog.identifier | Identifier (usually process) contained in the syslog header. | keyword |
| log.syslog.pid | PID contained in the syslog header. | long |
| log.syslog.priority | Syslog numeric priority of the event, if available. According to RFCs 5424 and 3164, the priority is 8 \* facility + severity. This number is therefore expected to contain a value between 0 and 191. | long |
| message | For log events the message field contains the log message, optimized for viewing in a log viewer. For structured logs without an original message field, other fields can be concatenated to form a human-readable summary of the event. If multiple messages exist, they can be combined into one message. | text |
| message | For log events the message field contains the log message, optimized for viewing in a log viewer. For structured logs without an original message field, other fields can be concatenated to form a human-readable summary of the event. If multiple messages exist, they can be combined into one message. | match_only_text |
| process.args | Array of process arguments, starting with the absolute path to the executable. May be filtered to protect sensitive information. | keyword |
| process.args_count | Length of the process.args array. This field can be useful for querying or performing bucket analysis on how many arguments were provided to start a process. More arguments may be an indication of suspicious activity. | long |
| process.command_line | Full command line that started the process, including the absolute path to the executable, and all arguments. Some arguments may be filtered to protect sensitive information. | keyword |
| process.command_line.text | Multi-field of `process.command_line`. | text |
| process.command_line | Full command line that started the process, including the absolute path to the executable, and all arguments. Some arguments may be filtered to protect sensitive information. | wildcard |
| process.command_line.text | Multi-field of `process.command_line`. | match_only_text |
| process.pid | Process id. | long |
| systemd.cgroup | The control group path in the systemd hierarchy. | keyword |
| systemd.invocation_id | The invocation ID for the runtime cycle of the unit the message was generated in, as available to processes of the unit in $INVOCATION_ID. | keyword |
Expand All @@ -74,4 +182,3 @@ The journald input is available on Linux systems with `systemd` installed.
| tags | List of keywords used to tag each event. | keyword |
| user.group.id | Unique identifier for the group on the system/platform. | keyword |
| user.id | Unique identifier of the user. | keyword |

4 changes: 2 additions & 2 deletions packages/journald/manifest.yml
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
format_version: 1.0.0
name: journald
title: "Custom Journald logs"
version: 0.0.3
version: 0.0.4
license: basic
description: Collect logs from journald with Elastic Agent.
type: integration
categories:
- custom
release: experimental
conditions:
kibana.version: "^7.16.0"
kibana.version: "^7.17.0 || ^8.1.0"
icons:
- src: /img/systemd-logo.svg
title: systemd logo
Expand Down