Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Documenting metrics messages in Beats logs #36163

Merged
merged 16 commits into from
Aug 1, 2023
Merged

Documenting metrics messages in Beats logs #36163

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
ad6112c
Documenting metrics messages in Beats logs
ycombinator Jul 26, 2023
252e92b
Add include statements & other structure
kilfoyle Jul 26, 2023
64cb9d2
Update libbeat/docs/metrics-in-logs.asciidoc
ycombinator Jul 29, 2023
054ab3a
Update libbeat/docs/metrics-in-logs.asciidoc
ycombinator Jul 29, 2023
a47b0b5
Update libbeat/docs/metrics-in-logs.asciidoc
ycombinator Jul 29, 2023
19002c9
Update libbeat/docs/metrics-in-logs.asciidoc
ycombinator Jul 29, 2023
f21bf3f
Update libbeat/docs/metrics-in-logs.asciidoc
ycombinator Jul 29, 2023
eca83f6
Update libbeat/docs/metrics-in-logs.asciidoc
ycombinator Jul 29, 2023
20a2e7b
Update libbeat/docs/metrics-in-logs.asciidoc
ycombinator Jul 29, 2023
26f328a
Update libbeat/docs/metrics-in-logs.asciidoc
ycombinator Jul 29, 2023
11ff8ac
Review feedback
ycombinator Jul 31, 2023
f7fc24d
Review feedback
ycombinator Jul 31, 2023
f2dcf63
Format JSON
ycombinator Jul 31, 2023
f52b188
Replace occurrences of Beat with variable
ycombinator Jul 31, 2023
c5b265b
Add link to unstructured logs' parsing script
ycombinator Jul 31, 2023
4e80569
Clarifying pipeline size metrics
ycombinator Jul 31, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions auditbeat/docs/troubleshooting.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ following tips:

* <<getting-help>>
* <<enable-{beatname_lc}-debugging>>
* <<understand-{beatname_lc}-logs>>
* <<faq>>

//sets block macro for getting-help.asciidoc included in next section
Expand All @@ -26,5 +27,15 @@ include::{libbeat-dir}/getting-help.asciidoc[]

include::{libbeat-dir}/debugging.asciidoc[]

//sets block macro for metrics-in-logs.asciidoc included in next section

[id="understand-{beatname_lc}-logs"]
[role="xpack"]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious: what does this line do?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it doesn't do anything, and rather it's just a leftover from when we had to identified X-pack content. I left it in when I copied the content over from the other docs page just because I'm not 100% sure it's not used anymore.

== Understand metrics in {beatname_uc} logs

++++
<titleabbrev>Understand logged metrics</titleabbrev>
++++

include::{libbeat-dir}/metrics-in-logs.asciidoc[]

11 changes: 11 additions & 0 deletions filebeat/docs/troubleshooting.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ following tips:

* <<getting-help>>
* <<enable-filebeat-debugging>>
* <<understand-{beatname_lc}-logs>>
* <<faq>>

//sets block macro for getting-help.asciidoc included in next section
Expand All @@ -26,5 +27,15 @@ include::{libbeat-dir}/getting-help.asciidoc[]

include::{libbeat-dir}/debugging.asciidoc[]

//sets block macro for metrics-in-logs.asciidoc included in next section

[id="understand-{beatname_lc}-logs"]
[role="xpack"]
== Understand metrics in {beatname_uc} logs

++++
<titleabbrev>Understand logged metrics</titleabbrev>
++++

include::{libbeat-dir}/metrics-in-logs.asciidoc[]

10 changes: 10 additions & 0 deletions heartbeat/docs/troubleshooting.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ following tips:

* <<getting-help>>
* <<enable-heartbeat-debugging>>
* <<understand-{beatname_lc}-logs>>
* <<faq>>

//sets block macro for getting-help.asciidoc included in next section
Expand All @@ -26,5 +27,14 @@ include::{libbeat-dir}/getting-help.asciidoc[]

include::{libbeat-dir}/debugging.asciidoc[]

//sets block macro for metrics-in-logs.asciidoc included in next section

[id="understand-{beatname_lc}-logs"]
[role="xpack"]
== Understand metrics in {beatname_uc} logs

++++
<titleabbrev>Understand logged metrics</titleabbrev>
++++

include::{libbeat-dir}/metrics-in-logs.asciidoc[]
17 changes: 12 additions & 5 deletions .../troubleshooting/metrics-in-logs.asciidoc → libbeat/docs/metrics-in-logs.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
== Understand metrics in {beatname_uc} logs

=== Introduction

Every 30 seconds (by default), Beats collect a _snapshot_ of metrics about itself. From this snapshot, Beat computes a _delta snapshot_; this delta snapshot contains any metrics that have _changed_ since the last snapshot. Note that the values of the metrics are the values when the snapshot is taken, **NOT** the _difference_ in values from the last snapshot.

ycombinator marked this conversation as resolved.
Show resolved Hide resolved
Expand All @@ -11,7 +9,8 @@ If this delta snapshot contains _any_ metrics (indicating that at least one metr
{"log.level":"info","@timestamp":"2023-07-14T12:50:36.811Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":0}}}},"cpu":{"system":{"ticks":692690,"time":{"ms":60}},"total":{"ticks":3167250,"time":{"ms":150},"value":3167250},"user":{"ticks":2474560,"time":{"ms":90}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":32},"info":{"ephemeral_id":"2bab8688-34c0-4522-80af-db86948d547d","uptime":{"ms":617670096},"version":"8.6.2"},"memstats":{"gc_next":57189272,"memory_alloc":43589824,"memory_total":275281335792,"rss":183574528},"runtime":{"goroutines":212}},"filebeat":{"events":{"active":5,"added":52,"done":49},"harvester":{"open_files":6,"running":6,"started":1}},"libbeat":{"config":{"module":{"running":15}},"output":{"events":{"acked":48,"active":0,"batches":6,"total":48},"read":{"bytes":210},"write":{"bytes":26923}},"pipeline":{"clients":15,"events":{"active":5,"filtered":1,"published":51,"total":52},"queue":{"acked":48}}},"registrar":{"states":{"current":14,"update":49},"writes":{"success":6,"total":6}},"system":{"load":{"1":0.91,"15":0.37,"5":0.4,"norm":{"1":0.1138,"15":0.0463,"5":0.05}}}},"ecs.version":"1.6.0"}}
----

=== Details
[discrete]
== Details

Focussing on the `.monitoring.metrics` field, it's value is:

Expand All @@ -28,7 +27,9 @@ The following tables attempt to explain the meaning of salient fields under `.mo

| `.beat` | Object | Information that is common to all Beats, e.g. version, goroutines, file handles, CPU, memory |
| `.libbeat` | Object | Information about the publisher pipeline and output, also common to all Beats |
ifeval::["{beatname_lc}"=="filebeat"]
| `.filebeat` | Object | Information specific to Filebeat, e.g. harvester, events |
endif::[]
ycombinator marked this conversation as resolved.
Show resolved Hide resolved
ycombinator marked this conversation as resolved.
Show resolved Hide resolved
|===

[cols="1,1,2,2"]
Expand All @@ -48,16 +49,21 @@ The following tables attempt to explain the meaning of salient fields under `.mo
| `.output.events.failed` | Integer | Number of events that the Beat tried to send to the output destination but it failed to receive them. | Generally, we want this field to be absent or its value to be 0. When the value is greater than zero, it's useful to check the Beat's logs right before this log entry's `@timestamp` to see if there are any connectivity issues with the output destination. Note that failed events are not lost or dropped; they will be sent back to the publisher pipeline for retrying later.
|===
ycombinator marked this conversation as resolved.
Show resolved Hide resolved

ifeval::["{beatname_lc}"=="filebeat"]
[cols="1,1,2,2"]
|===
| Field path (relative to `.monitoring.metrics.filebeat`) | Type | Meaning | Troubleshooting hints

| `.events.active` | Integer | Number of events being actively processed by Filebeat (including events Filebeat has already sent to the libbeat publisher pipeline, but not including events the pipeline has sent to the output). | If this number grows over time, it may indicate that Filebeat inputs are harvesting events too fast for the pipeline and output to keep up.
|===
ycombinator marked this conversation as resolved.
Show resolved Hide resolved
endif::[]

=== Useful commands
ifeval::["{beatname_lc}"=="filebeat"]
[discrete]
== Useful commands

==== Check if Filebeat is processing events
[discrete]
=== Check if Filebeat is processing events

[source]
----
Expand Down Expand Up @@ -103,3 +109,4 @@ The columns here are:
4. `.monitoring.metrics.libbeat.output.events.total`
5. `.monitoring.metrics.libbeat.output.events.acked`
6. `.monitoring.metrics.libbeat.output.events.failed`
endif::[]
13 changes: 13 additions & 0 deletions metricbeat/docs/troubleshooting.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ If you have issues installing or running {beatname_uc}, read the following tips:

* <<getting-help>>
* <<enable-metricbeat-debugging>>
* <<understand-{beatname_lc}-logs>>
* <<faq>>

//sets block macro for getting-help.asciidoc included in next section
Expand All @@ -24,3 +25,15 @@ include::{libbeat-dir}/getting-help.asciidoc[]
== Debug

include::{libbeat-dir}/debugging.asciidoc[]

//sets block macro for metrics-in-logs.asciidoc included in next section

[id="understand-{beatname_lc}-logs"]
[role="xpack"]
== Understand metrics in {beatname_uc} logs

++++
<titleabbrev>Understand logged metrics</titleabbrev>
++++

include::{libbeat-dir}/metrics-in-logs.asciidoc[]
13 changes: 13 additions & 0 deletions packetbeat/docs/troubleshooting.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ following tips:

* <<getting-help>>
* <<enable-packetbeat-debugging>>
* <<understand-{beatname_lc}-logs>>
* <<recording-trace>>
* <<faq>>

Expand All @@ -29,6 +30,18 @@ include::{libbeat-dir}/debugging.asciidoc[]

//sets block macro for recording-trace content included in next section

//sets block macro for metrics-in-logs.asciidoc included in next section

[id="understand-{beatname_lc}-logs"]
[role="xpack"]
== Understand metrics in {beatname_uc} logs

++++
<titleabbrev>Understand logged metrics</titleabbrev>
++++

include::{libbeat-dir}/metrics-in-logs.asciidoc[]

[[recording-trace]]
== Record a trace

Expand Down
13 changes: 13 additions & 0 deletions winlogbeat/docs/troubleshooting.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ If you have issues installing or running Winlogbeat, read the following tips:

* <<getting-help>>
* <<enable-winlogbeat-debugging>>
* <<understand-{beatname_lc}-logs>>
* <<faq>>

//sets block macro for getting-help.asciidoc included in next section
Expand All @@ -25,3 +26,15 @@ include::{libbeat-dir}/getting-help.asciidoc[]
== Debug

include::{libbeat-dir}/debugging.asciidoc[]

//sets block macro for metrics-in-logs.asciidoc included in next section

[id="understand-{beatname_lc}-logs"]
[role="xpack"]
== Understand metrics in {beatname_uc} logs

++++
<titleabbrev>Understand logged metrics</titleabbrev>
++++

include::{libbeat-dir}/metrics-in-logs.asciidoc[]
10 changes: 10 additions & 0 deletions x-pack/functionbeat/docs/troubleshooting.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ following tips:

* <<getting-help>>
* <<enable-{beatname_lc}-debugging>>
* <<understand-{beatname_lc}-logs>>
* <<faq>>

//sets block macro for getting-help.asciidoc included in next section
Expand All @@ -29,5 +30,14 @@ include::{libbeat-dir}/getting-help.asciidoc[]

include::{libbeat-dir}/debugging.asciidoc[]

//sets block macro for metrics-in-logs.asciidoc included in next section

[id="understand-{beatname_lc}-logs"]
[role="xpack"]
== Understand metrics in {beatname_uc} logs

++++
<titleabbrev>Understand logged metrics</titleabbrev>
++++

include::{libbeat-dir}/metrics-in-logs.asciidoc[]
Loading