Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Metricbeat] Fix fields not being parsed correctly in postgresql/database #29051

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Fix list_docker.go {pull}28374[28374]
- Use xpack.enabled on SM modules to write into .monitoring indices when using Metricbeat standalone {pull}28365[28365]
- Fix in rename processor to ingest metrics for `write.iops` to proper field instead of `write_iops` in rds metricset. {pull}28960[28960]
- Fix fields not being parsed correctly in postgresql/database {issue}25301[25301] {pull}29051[29051]

*Packetbeat*

Expand Down
4 changes: 2 additions & 2 deletions metricbeat/docs/fields.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -53948,7 +53948,7 @@ type: long
Time spent reading data file blocks by backends in this database, in milliseconds.


type: long
type: double

--

Expand All @@ -53958,7 +53958,7 @@ type: long
Time spent writing data file blocks by backends in this database, in milliseconds.


type: long
type: double

--

Expand Down
82 changes: 50 additions & 32 deletions metricbeat/module/postgresql/database/_meta/data.json
Original file line number Diff line number Diff line change
@@ -1,53 +1,71 @@
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"@timestamp": "2021-11-19T13:08:37.241Z",
"@metadata": {
"beat": "metricbeat",
"type": "_doc",
"version": "8.1.0"
},
"service": {
"type": "postgresql",
"address": "postgres://localhost:5432/test?connect_timeout=10&sslmode=disable"
},
"event": {
"duration": 12127879,
"dataset": "postgresql.database",
"duration": 115000,
"module": "postgresql"
},
"metricset": {
"name": "database",
"period": 10000
},
"postgresql": {
"database": {
"deadlocks": 0,
"stats_reset": "2021-11-19T13:03:43.827Z",
"temporary": {
"files": 0,
"bytes": 0
},
"transactions": {
"rollback": 5,
"commit": 141
},
"blocks": {
"hit": 43969,
"read": 381,
"time": {
"read": {
"ms": 0
"ms": 7.686
},
"write": {
"ms": 0
}
}
},
"read": 726,
"hit": 26718
},
"conflicts": 0,
"deadlocks": 0,
"name": "postgres",
"number_of_backends": 2,
"oid": 13395,
"number_of_backends": 4,
"oid": 16384,
"name": "test",
"rows": {
"deleted": 42,
"fetched": 26418,
"inserted": 98,
"returned": 53201,
"updated": 14
},
"stats_reset": "2021-03-05T18:39:18.089Z",
"temporary": {
"bytes": 0,
"files": 0
},
"transactions": {
"commit": 445,
"rollback": 5
"deleted": 31,
"returned": 46331,
"fetched": 13006,
"inserted": 71,
"updated": 2
}
}
},
"service": {
"address": "192.168.128.2:5432",
"type": "postgresql"
"ecs": {
"version": "8.0.0"
},
"host": {
"name": "thinkpad"
},
"agent": {
"type": "metricbeat",
"version": "8.1.0",
"ephemeral_id": "ce27b375-714a-4304-9b41-5f1cba028d6e",
"id": "7aaa6d29-9700-452a-bd12-2bdd2a2f884e",
"name": "thinkpad"
},
"metricset": {
"name": "database",
"period": 10000
}
}
}
4 changes: 2 additions & 2 deletions metricbeat/module/postgresql/database/_meta/fields.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,12 +36,12 @@
that a read was not necessary (this only includes hits in the PostgreSQL
buffer cache, not the operating system's file system cache).
- name: blocks.time.read.ms
type: long
type: double
description: >
Time spent reading data file blocks by backends in this database, in
milliseconds.
- name: blocks.time.write.ms
type: long
type: double
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, double check if this change is problematic in a deployment with both versions of Metricbeat, we had recently some bad experiences with type changes that looked safe.

The kind of test you could do is:

  • Install Metricbeat 7.15 and create a visualization using this field (even if it is zero).
  • Double-check that you have a metricbeat-* index pattern where this field appears.
  • In the same deployment, update to Metricbeat with this fix, and check if there is any problem with the visualization or with the index pattern.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah please double check here 😂 Not sure if it's the same case but I accidentally made a breaking change for changing field type from scaled float to long.

description: >
Time spent writing data file blocks by backends in this database, in
milliseconds.
Expand Down
4 changes: 2 additions & 2 deletions metricbeat/module/postgresql/database/data.go
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,8 @@ var schema = s.Schema{
"read": c.Int("blks_read"),
"hit": c.Int("blks_hit"),
"time": s.Object{
"read": s.Object{"ms": c.Int("blk_read_time")},
"write": s.Object{"ms": c.Int("blk_write_time")},
"read": s.Object{"ms": c.Float("blk_read_time")},
"write": s.Object{"ms": c.Float("blk_write_time")},
},
},
"rows": s.Object{
Expand Down
2 changes: 1 addition & 1 deletion metricbeat/module/postgresql/fields.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.