Elasticsearch server fileset processes logs of other metricsets

[jsoriano]

Default paths configuration in the Elasticsearch server metricset is too broad, so it harvests files that should be harvested by other metricsets, resulting in incorrectly parsed logs.

Default paths in the server metricset include patterns like /var/log/elasticsearch/*.log.
Logs that should be managed by other filesets match this pattern, e.g. the GC one: /var/log/elasticsearch/gc.log, or the deprecation one: /var/log/elasticsearch/some-name_deprecation.log.
So it can happen that the server fileset harvests these files, selecting for them an incorrect pipeline.

---

While this is fixed, a possible workaround is to set the paths in the server metricset to the actual server logs, something like this:

- module: elasticsearch
  server:
    enabled: true
    var.paths:
      - '/var/log/elasticsearch/some-name.log'
      - '/var/log/elasticsearch/some-name.log.*'
  ...

Or for JSON logs:

- module: elasticsearch
  server:
    enabled: true
    var.paths:
      - '/var/log/elasticsearch/*_server.json'
      - '/var/log/elasticsearch/*_server.json.*'
  ...

---

• [Filebeat] Elasticsearch Module w/ Kubernetes Autodiscover Causes Logs in Incorrect Fieldsets #16540 can be related

[elasticmachine]

Pinging @elastic/stack-monitoring (Stack monitoring)

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[milkfinch]

We have the same error for deprecation logs (strangely not for all of the others) after we set up the above workaround.

After we specify the proper log files for all the different sections for "var.paths" the problem was solved.

[jsoriano]

We have the same error for deprecation logs (strangely not for all of the others) after we set up the above workaround.

Interesting, what was the config after applying the workaround? In what path were the deprecation logs?

[botelastic]

Hi!
We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!