[Filebeat] Elasticsearch Module w/ Kubernetes Autodiscover Causes Logs in Incorrect Fieldsets #16540

SpencerLN · 2020-02-25T05:07:31Z

Issue: Using the Filebeat Elasticsearch module in combination with Kubernetes autodiscover results in logs in the incorrect filesets or duplicate filesets:

Expected behavior: Each log message should only appear in the destination a single time, and it should have the appropriate fields associated with the fileset of that log (i.e. server, audit, deprecation, gc, etc.)

Version: 7.6.0
Operating System: GKE Container Optimized OS
Steps to Reproduce:
Deploying Filebeat using Filebeat Helm chart with values.yaml:

filebeat: 
  # Allows you to add any config files in /usr/share/filebeat
  # such as filebeat.yml
  filebeatConfig:
    filebeat.yml: |-
      filebeat.autodiscover:
       providers:
         - type: kubernetes
           node: ${NODE_NAME}
           hints.enabled: true

      processors:
        - add_cloud_metadata:
        - add_host_metadata:

      output.elasticsearch:
          hosts: 'http://${ELASTICSEARCH_ADDRESS}'
          username: ${ELASTICSEARCH_USERNAME}
          password: ${ELASTICSEARCH_PASSWORD}

  extraEnvs:
    - name: ELASTICSEARCH_USERNAME
      valueFrom: { secretKeyRef: { name: beats-elasticsearch, key: username } }
    - name: ELASTICSEARCH_PASSWORD
      valueFrom: { secretKeyRef: { name: beats-elasticsearch, key: password } }
    - name: ELASTICSEARCH_ADDRESS
      valueFrom: { secretKeyRef: { name: beats-elasticsearch, key: address } }

  imageTag: 7.6.0
  extraVolumeMounts: 
    - name: varlog
      mountPath: /var/log/
      readOnly: true

  extraVolumes: 
    - name: varlog
      hostPath:
        path: /var/log/

In this case Elasticsearch was deployed using ECK, with annotations in the podTemplate to enable the Elasticsearch module:

    podTemplate:
      metadata:
        annotations:
          co.elastic.logs/module: "elasticsearch"

I am attaching the filebeat.log file, the json of the documents from the above screenshot, and the contents of the bulk request that show the documents are being to the corresponding pipelines for each fileset (4 copies of the same document).

The appropriate pipelines were installed in Elasticsearch prior to these logs being ingested:

filebeat.log
bulk_request.txt
sample-docs.txt

The text was updated successfully, but these errors were encountered:

ycombinator · 2020-02-25T12:27:11Z

As a first step it would be good to try to reproduce this with Elasticsearch simply running in a Docker container and pointing Filebeat to it's logs. That is, if we can reproduce this without k8s or autodisovery in the picture, it might let us narrow down the source of the problem.

blakerouse · 2020-02-25T14:22:58Z

Can you try 7.x or master? I believe you might get a different behavior with this change #16450, but I could be wrong. I remember when working on that change that multiple filesets where getting merged.

I think the issue here is that autodiscovery only reads from the container log, which is a single log file, it will not read from logs inside of the container. With merging the paths are replaced with the container log path and not the actual path inside of the container.

SpencerLN · 2020-02-25T16:21:01Z

@ycombinator I can spin up an Elasticsearch node and point Filebeat to it's logs, but I am having trouble figuring out what the filebeat.yml config file to accomplish that would look like without using autodiscover.

Is this what you are suggesting for the test?

filebeat.yml:

filebeat.inputs:
- type: log
  enabled: false
  paths:
    - /var/log/*.log
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
output.elasticsearch:
  hosts: ["https://elasticsearch:9200"]
  username: "username"
  password: "password"

modules.d/elasticsearch.yml:

- module: elasticsearch
  # Server log
  server:
    input:
      type: container
      paths:
	- '/var/lib/docker/containers/*/*.log'
    enabled: true

  gc:
    input:
      type: container
      paths:
	- '/var/lib/docker/containers/*/*.log'
    enabled: true

  audit:
    input:
      type: container
      paths:
	- '/var/lib/docker/containers/*/*.log'
    enabled: true

  slowlog:
    input:
      type: container
      paths:
	- '/var/lib/docker/containers/*/*.log'
    enabled: true

  deprecation:
    input:
      type: container
      paths:
	- '/var/lib/docker/containers/*/*.log'
    enabled: true

SpencerLN · 2020-02-25T18:04:06Z

Ok, I did some testing with the above config and got better results than I was seeing in Kubernetes. The only fileset that ended up with unexpected data was the gc fileset:

Log output from yml based module config:

Feb 25 17:41:17 spencer-docker-test-rhel7 filebeat[9743]: 2020-02-25T17:41:17.751Z        INFO        log/input.go:152        Configured paths: [/var/lib/docker/containers/*/*.log /var/log/elasticsearch/gc.log]
Feb 25 17:41:17 spencer-docker-test-rhel7 filebeat[9743]: 2020-02-25T17:41:17.759Z        INFO        log/input.go:152        Configured paths: [/var/lib/docker/containers/*/*.log /var/log/elasticsearch/*_server.json]
Feb 25 17:41:17 spencer-docker-test-rhel7 filebeat[9743]: 2020-02-25T17:41:17.767Z        INFO        log/input.go:152        Configured paths: [/var/lib/docker/containers/*/*.log /var/log/elasticsearch/*_index_indexing_slowlog.log /var/log/elasticsearch/*_index_search_slowlog.json /var/log/elasticsearch/*_index_indexing_slowlog.json]
Feb 25 17:41:17 spencer-docker-test-rhel7 filebeat[9743]: 2020-02-25T17:41:17.773Z        INFO        log/input.go:152        Configured paths: [/var/lib/docker/containers/*/*.log /var/log/elasticsearch/*_audit.log /var/log/elasticsearch/*_audit.json]
Feb 25 17:41:17 spencer-docker-test-rhel7 filebeat[9743]: 2020-02-25T17:41:17.780Z        INFO        log/input.go:152        Configured paths: [/var/lib/docker/containers/*/*.log /var/log/elasticsearch/*_deprecation.json]

I then tested using the docker autodiscover provider:

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

Log output from docker autodiscover config:

Feb 25 17:03:41 spencer-docker-test-rhel7 filebeat[7830]: 2020-02-25T17:03:41.382Z        INFO        log/input.go:152        Configured paths: [/var/lib/docker/containers/8ac09baaa513a56e4a6e7d10dbf42324936ee129dfaa645cf87ca3e40ff5974d/*-json.log /var/log/elasticsearch/*_audit.log /var/log/elasticsearch/*_audit.json]
Feb 25 17:03:41 spencer-docker-test-rhel7 filebeat[7830]: 2020-02-25T17:03:41.383Z        INFO        log/input.go:152        Configured paths: [/var/lib/docker/containers/8ac09baaa513a56e4a6e7d10dbf42324936ee129dfaa645cf87ca3e40ff5974d/*-json.log /var/log/elasticsearch/*_deprecation.json]
Feb 25 17:03:41 spencer-docker-test-rhel7 filebeat[7830]: 2020-02-25T17:03:41.384Z        INFO        log/input.go:152        Configured paths: [/var/lib/docker/containers/8ac09baaa513a56e4a6e7d10dbf42324936ee129dfaa645cf87ca3e40ff5974d/*-json.log /var/log/elasticsearch/gc.log]
Feb 25 17:03:41 spencer-docker-test-rhel7 filebeat[7830]: 2020-02-25T17:03:41.384Z        INFO        log/input.go:152        Configured paths: [/var/lib/docker/containers/8ac09baaa513a56e4a6e7d10dbf42324936ee129dfaa645cf87ca3e40ff5974d/*-json.log /var/log/elasticsearch/*_server.json]
Feb 25 17:03:41 spencer-docker-test-rhel7 filebeat[7830]: 2020-02-25T17:03:41.385Z        INFO        log/input.go:152        Configured paths: [/var/lib/docker/containers/8ac09baaa513a56e4a6e7d10dbf42324936ee129dfaa645cf87ca3e40ff5974d/*-json.log /var/log/elasticsearch/*_index_indexing_slowlog.log /var/log/elasticsearch/*_index_search_slowlog.json /var/log/elasticsearch/*_ind
ex_indexing_slowlog.json]

For the Docker autodiscover provider I got the same results as I did with manually configuring the module in the previous test, where only the gc fileset contained data that it shouldn't.

Taking this same config, I deployed it to Kubernetes:

filebeat.autodiscover:
 providers:
   - type: kubernetes
     node: ${NODE_NAME}
     hints.enabled: true

This resulted in again seeing log entries in random filesets in Elasticsearch:

Log output from kubernetes autodiscover config:

2020-02-25T17:31:42.276Z	INFO	log/input.go:152	Configured paths: [/var/lib/docker/containers/0cb79921b0530ae342e032560da511e1478a2ff6aa02d01a69038c1fdfe93da8/*-json.log /var/log/elasticsearch/*_index_indexing_slowlog.log /var/log/elasticsearch/*_index_search_slowlog.json /var/log/elasticsearch/*_index_indexing_slowlog.json]
2020-02-25T17:31:42.276Z	INFO	log/input.go:152	Configured paths: [/var/lib/docker/containers/0cb79921b0530ae342e032560da511e1478a2ff6aa02d01a69038c1fdfe93da8/*-json.log /var/log/elasticsearch/*_audit.log /var/log/elasticsearch/*_audit.json]
2020-02-25T17:31:42.277Z	INFO	log/input.go:152	Configured paths: [/var/lib/docker/containers/0cb79921b0530ae342e032560da511e1478a2ff6aa02d01a69038c1fdfe93da8/*-json.log /var/log/elasticsearch/*_deprecation.json]
2020-02-25T17:31:42.278Z	INFO	log/input.go:152	Configured paths: [/var/lib/docker/containers/0cb79921b0530ae342e032560da511e1478a2ff6aa02d01a69038c1fdfe93da8/*-json.log /var/log/elasticsearch/gc.log]
2020-02-25T17:31:42.278Z	INFO	log/input.go:152	Configured paths: [/var/lib/docker/containers/0cb79921b0530ae342e032560da511e1478a2ff6aa02d01a69038c1fdfe93da8/*-json.log /var/log/elasticsearch/*_server.json]

In all 3 cases, the GC logs were an issue and logs that were not GC logs ended up in the GC fileset. Only the Kubernetes autodiscover test resulted in the additional issues where we saw the log entries erroneously appear in the deprecation, audit, server, etc. filesets.

ycombinator · 2020-02-26T00:11:55Z

Thanks for the superbly detailed breakdown, @SpencerLN — appreciate it!

So it seems like there are two distinct issues here we need to address:

Teach the elasticsearch/gc fileset to only ingest GC logs. Filed [Filebeat] Elasticsearch gc fileset collects non-GC logs in container environment #16583 to separately track and resolve this.
Look into why k8s autodiscover is causing random logs to "crossover" to other Elasticsearch module filesets. For this one, it might be good to also understand if this behavior is specific to the Elasticsearch module or if it happens with any Filebeat modules.

SpencerLN · 2020-02-26T02:19:03Z

@ycombinator I did some additional testing tonight and found that if I disable the default_config for Kubernetes autodiscover, I no longer see the unexpected behavior, and have identical results as I did in Docker.

      filebeat.autodiscover:
       providers:
         - type: kubernetes
           node: ${NODE_NAME}
           hints.enabled: true
           hints.default_config.enabled: false

Unfortunately, this isn't an ideal configuration, since we do want to collect logs from Pods that we don't necessarily control and cannot always annotate. That said, hopefully it helps in narrowing the cause of the issue.

We are also running the Kibana module via Kubernetes autodiscover and don't seem to see the same problems there, but the Kibana module only has one fileset, so I am not sure if that is a factor or not.

Let me know if there is more verbose logging level for a certain component of filebeat that would provide additional helpful information.

Thanks!

ycombinator · 2020-02-26T02:25:17Z

Thanks again @SpencerLN, this does help narrow it down!

@blakerouse Do you want to tackle the k8s bits or file an issue for it and reference it in #16540 (comment)? Then we can treat this issue here as a meta issue for the two separate issues referenced from that comment.

blakerouse · 2020-02-26T14:38:41Z

I think this bug covers the issue for the autodiscover, so lets keep this issue open.

nerddelphi · 2020-03-02T01:46:57Z

@blakerouse and @ycombinator
I guess I have a related issue using HAProxy module, please check it: #16464 (comment)

blakerouse · 2020-03-10T19:51:44Z

Been looking into this, attaching debug log so I do not lose it.

Debug log for autodiscover, config, and kubernetes from filebeat:7.7.0-SNAPSHOT. Shows the merged configs with autodiscovery on kubernetes and the elasticsearch module.

2020-03-10T18:12:35.038Z	DEBUG	[autodiscover]	autodiscover/autodiscover.go:190	Generated config: map[audit:map[enabled:true input:map[paths:[/var/log/containers/*1bc7e3d9262e847422ace7ef946c39e59abcca679de472207a4af0734a5a859a.log] stream:all type:container]] deprecation:map[enabled:true input:map[paths:[/var/log/containers/*1bc7e3d9262e847422ace7ef946c39e59abcca679de472207a4af0734a5a859a.log] stream:all type:container]] gc:map[enabled:true input:map[paths:[/var/log/containers/*1bc7e3d9262e847422ace7ef946c39e59abcca679de472207a4af0734a5a859a.log] stream:all type:container]] module:elasticsearch server:map[enabled:true input:map[paths:[/var/log/containers/*1bc7e3d9262e847422ace7ef946c39e59abcca679de472207a4af0734a5a859a.log] stream:all type:container]] slowlog:map[enabled:true input:map[paths:[/var/log/containers/*1bc7e3d9262e847422ace7ef946c39e59abcca679de472207a4af0734a5a859a.log] stream:all type:container]]]
2020-03-10T18:12:35.038Z	DEBUG	[autodiscover]	autodiscover/autodiscover.go:252	Got a meta field in the event
2020-03-10T18:12:35.038Z	DEBUG	[autodiscover]	cfgfile/list.go:62	Starting reload procedure, current runners: 4
2020-03-10T18:12:35.038Z	DEBUG	[autodiscover]	cfgfile/list.go:80	Start list: 1, Stop list: 0
2020-03-10T18:12:35.041Z	DEBUG	[config]	common/config.go:276	load config file '/usr/share/filebeat/module/elasticsearch/audit/manifest.yml' =>
{
  "ingest_pipeline": [
    "ingest/pipeline.json",
    "ingest/pipeline-json.json",
    "ingest/pipeline-plaintext.json"
  ],
  "input": "config/audit.yml",
  "module_version": 1,
  "var": [
    {
      "default": [
        "/var/log/elasticsearch/*_access.log",
        "/var/log/elasticsearch/*_audit.log",
        "/var/log/elasticsearch/*_audit.json"
      ],
      "name": "paths",
      "os": {
        "darwin": [
          "/usr/local/var/lib/elasticsearch/*_access.log",
          "/usr/local/var/lib/elasticsearch/*_audit.log",
          "/usr/local/var/lib/elasticsearch/*_audit.json"
        ],
        "windows": [
          "c:/ProgramData/Elastic/Elasticsearch/logs/*_access.log",
          "c:/ProgramData/Elastic/Elasticsearch/logs/*_audit.log",
          "c:/ProgramData/Elastic/Elasticsearch/logs/*_audit.json"
        ]
      }
    }
  ]
}
2020-03-10T18:12:35.042Z	DEBUG	[config]	common/config.go:276	load config file '/usr/share/filebeat/module/elasticsearch/deprecation/manifest.yml' =>
{
  "ingest_pipeline": [
    "ingest/pipeline.json",
    "ingest/pipeline-plaintext.json",
    "ingest/pipeline-json.json"
  ],
  "input": "config/log.yml",
  "module_version": 1,
  "var": [
    {
      "default": [
        "/var/log/elasticsearch/*_deprecation.log",
        "/var/log/elasticsearch/*_deprecation.json"
      ],
      "name": "paths",
      "os": {
        "darwin": [
          "/usr/local/var/lib/elasticsearch/*_deprecation.log",
          "/usr/local/var/lib/elasticsearch/*_deprecation.json"
        ],
        "windows": [
          "c:/ProgramData/Elastic/Elasticsearch/logs/*_deprecation.log",
          "c:/ProgramData/Elastic/Elasticsearch/logs/*_deprecation.json"
        ]
      }
    }
  ]
}
2020-03-10T18:12:35.043Z	DEBUG	[config]	common/config.go:276	load config file '/usr/share/filebeat/module/elasticsearch/gc/manifest.yml' =>
{
  "ingest_pipeline": "ingest/pipeline.json",
  "input": "config/gc.yml",
  "module_version": 1,
  "var": [
    {
      "default": [
        "/var/log/elasticsearch/gc.log.[0-9]*",
        "/var/log/elasticsearch/gc.log"
      ],
      "name": "paths",
      "os": {
        "darwin": [
          "/usr/local/var/lib/elasticsearch/gc.log.[0-9]*",
          "/usr/local/var/lib/elasticsearch/gc.log"
        ],
        "windows": [
          "c:/ProgramData/Elastic/Elasticsearch/logs/gc.log.*",
          "c:/ProgramData/Elastic/Elasticsearch/logs/gc.log"
        ]
      }
    }
  ]
}
2020-03-10T18:12:35.043Z	DEBUG	[config]	common/config.go:276	load config file '/usr/share/filebeat/module/elasticsearch/server/manifest.yml' =>
{
  "ingest_pipeline": [
    "ingest/pipeline.json",
    "ingest/pipeline-plaintext.json",
    "ingest/pipeline-json.json"
  ],
  "input": "config/log.yml",
  "module_version": 1,
  "var": [
    {
      "default": [
        "/var/log/elasticsearch/*.log",
        "/var/log/elasticsearch/*_server.json"
      ],
      "name": "paths",
      "os": {
        "darwin": [
          "/usr/local/var/lib/elasticsearch/*.log",
          "/usr/local/var/lib/elasticsearch/*_server.json"
        ],
        "windows": [
          "c:/ProgramData/Elastic/Elasticsearch/logs/*.log",
          "c:/ProgramData/Elastic/Elasticsearch/logs/*_server.json"
        ]
      }
    }
  ]
}
2020-03-10T18:12:35.044Z	DEBUG	[config]	common/config.go:276	load config file '/usr/share/filebeat/module/elasticsearch/slowlog/manifest.yml' =>
{
  "ingest_pipeline": [
    "ingest/pipeline.json",
    "ingest/pipeline-plaintext.json",
    "ingest/pipeline-json.json"
  ],
  "input": "config/slowlog.yml",
  "module_version": 1,
  "var": [
    {
      "default": [
        "/var/log/elasticsearch/*_index_search_slowlog.log",
        "/var/log/elasticsearch/*_index_indexing_slowlog.log",
        "/var/log/elasticsearch/*_index_search_slowlog.json",
        "/var/log/elasticsearch/*_index_indexing_slowlog.json"
      ],
      "name": "paths",
      "os": {
        "darwin": [
          "/usr/local/var/lib/elasticsearch/*_index_search_slowlog.log",
          "/usr/local/var/lib/elasticsearch/*_index_indexing_slowlog.log",
          "/usr/local/var/lib/elasticsearch/*_index_search_slowlog.json",
          "/usr/local/var/lib/elasticsearch/*_index_indexing_slowlog.json"
        ],
        "windows": [
          "c:/ProgramData/Elastic/Elasticsearch/logs/*_index_search_slowlog.log",
          "c:/ProgramData/Elastic/Elasticsearch/logs/*_index_indexing_slowlog.log",
          "c:/ProgramData/Elastic/Elasticsearch/logs/*_index_search_slowlog.json",
          "c:/ProgramData/Elastic/Elasticsearch/logs/*_index_indexing_slowlog.json"
        ]
      }
    }
  ]
}
2020-03-10T18:12:35.044Z	DEBUG	[config]	common/config.go:276	Merged input config for fileset elasticsearch/audit
{
  "_fileset_name": "audit",
  "_module_name": "elasticsearch",
  "exclude_files": [
    ".gz$"
  ],
  "path": {
    "config": "/usr/share/filebeat",
    "data": "/usr/share/filebeat/data",
    "home": "/usr/share/filebeat",
    "logs": "/usr/share/filebeat/logs"
  },
  "paths": [
    "/var/log/containers/*1bc7e3d9262e847422ace7ef946c39e59abcca679de472207a4af0734a5a859a.log"
  ],
  "pipeline": "filebeat-7.7.0-elasticsearch-audit-pipeline",
  "processors": [
    {
      "add_locale": null
    }
  ],
  "stream": "all",
  "type": "container"
}
2020-03-10T18:12:35.045Z	DEBUG	[config]	common/config.go:276	Merged input config for fileset elasticsearch/deprecation
{
  "_fileset_name": "deprecation",
  "_module_name": "elasticsearch",
  "exclude_files": [
    ".gz$",
    "_slowlog.log$",
    "_access.log$"
  ],
  "multiline": {
    "match": "after",
    "negate": true,
    "pattern": "^(\\[[0-9]{4}-[0-9]{2}-[0-9]{2}|{)"
  },
  "path": {
    "config": "/usr/share/filebeat",
    "data": "/usr/share/filebeat/data",
    "home": "/usr/share/filebeat",
    "logs": "/usr/share/filebeat/logs"
  },
  "paths": [
    "/var/log/containers/*1bc7e3d9262e847422ace7ef946c39e59abcca679de472207a4af0734a5a859a.log"
  ],
  "pipeline": "filebeat-7.7.0-elasticsearch-deprecation-pipeline",
  "processors": [
    {
      "add_locale": {
        "when": {
          "not": {
            "regexp": {
              "message": "^{"
            }
          }
        }
      }
    }
  ],
  "stream": "all",
  "type": "container"
}
2020-03-10T18:12:35.045Z	DEBUG	[config]	common/config.go:276	Merged input config for fileset elasticsearch/gc
{
  "_fileset_name": "gc",
  "_module_name": "elasticsearch",
  "exclude_files": [
    ".gz$"
  ],
  "exclude_lines": [
    "^(OpenJDK|Java HotSpot).* Server VM ",
    "^CommandLine flags: ",
    "^Memory: "
  ],
  "multiline": {
    "match": "after",
    "negate": true,
    "pattern": "^\\[?[0-9]{4}-[0-9]{2}-[0-9]{2}"
  },
  "path": {
    "config": "/usr/share/filebeat",
    "data": "/usr/share/filebeat/data",
    "home": "/usr/share/filebeat",
    "logs": "/usr/share/filebeat/logs"
  },
  "paths": [
    "/var/log/containers/*1bc7e3d9262e847422ace7ef946c39e59abcca679de472207a4af0734a5a859a.log"
  ],
  "pipeline": "filebeat-7.7.0-elasticsearch-gc-pipeline",
  "stream": "all",
  "type": "container"
}
2020-03-10T18:12:35.046Z	DEBUG	[config]	common/config.go:276	Merged input config for fileset elasticsearch/server
{
  "_fileset_name": "server",
  "_module_name": "elasticsearch",
  "exclude_files": [
    ".gz$",
    "_slowlog.log$",
    "_access.log$",
    "_deprecation.log$"
  ],
  "multiline": {
    "match": "after",
    "negate": true,
    "pattern": "^(\\[[0-9]{4}-[0-9]{2}-[0-9]{2}|{)"
  },
  "path": {
    "config": "/usr/share/filebeat",
    "data": "/usr/share/filebeat/data",
    "home": "/usr/share/filebeat",
    "logs": "/usr/share/filebeat/logs"
  },
  "paths": [
    "/var/log/containers/*1bc7e3d9262e847422ace7ef946c39e59abcca679de472207a4af0734a5a859a.log"
  ],
  "pipeline": "filebeat-7.7.0-elasticsearch-server-pipeline",
  "processors": [
    {
      "add_locale": {
        "when": {
          "not": {
            "regexp": {
              "message": "^{"
            }
          }
        }
      }
    }
  ],
  "stream": "all",
  "type": "container"
}
2020-03-10T18:12:35.046Z	DEBUG	[config]	common/config.go:276	Merged input config for fileset elasticsearch/slowlog
{
  "_fileset_name": "slowlog",
  "_module_name": "elasticsearch",
  "exclude_files": [
    ".gz$"
  ],
  "multiline": {
    "match": "after",
    "negate": true,
    "pattern": "^(\\[?[0-9]{4}-[0-9]{2}-[0-9]{2}|{)"
  },
  "path": {
    "config": "/usr/share/filebeat",
    "data": "/usr/share/filebeat/data",
    "home": "/usr/share/filebeat",
    "logs": "/usr/share/filebeat/logs"
  },
  "paths": [
    "/var/log/containers/*1bc7e3d9262e847422ace7ef946c39e59abcca679de472207a4af0734a5a859a.log"
  ],
  "pipeline": "filebeat-7.7.0-elasticsearch-slowlog-pipeline",
  "processors": [
    {
      "add_locale": {
        "when": {
          "not": {
            "regexp": {
              "message": "^{"
            }
          }
        }
      }
    }
  ],
  "stream": "all",
  "type": "container"
}
2020-03-10T18:12:35.047Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*1bc7e3d9262e847422ace7ef946c39e59abcca679de472207a4af0734a5a859a.log]
2020-03-10T18:12:35.047Z	DEBUG	[config]	common/config.go:276	Configure processor action 'add_locale' with:
{
  "when": {
    "not": {
      "regexp": {
        "message": "^{"
      }
    }
  }
}
2020-03-10T18:12:35.048Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*1bc7e3d9262e847422ace7ef946c39e59abcca679de472207a4af0734a5a859a.log]
2020-03-10T18:12:35.048Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*1bc7e3d9262e847422ace7ef946c39e59abcca679de472207a4af0734a5a859a.log]
2020-03-10T18:12:35.049Z	DEBUG	[config]	common/config.go:276	Configure processor action 'add_locale' with:
{
  "when": {
    "not": {
      "regexp": {
        "message": "^{"
      }
    }
  }
}
2020-03-10T18:12:35.049Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*1bc7e3d9262e847422ace7ef946c39e59abcca679de472207a4af0734a5a859a.log]
2020-03-10T18:12:35.049Z	DEBUG	[config]	common/config.go:276	Configure processor action 'add_locale' with:
{
  "when": {
    "not": {
      "regexp": {
        "message": "^{"
      }
    }
  }
}
2020-03-10T18:12:35.050Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*1bc7e3d9262e847422ace7ef946c39e59abcca679de472207a4af0734a5a859a.log]
2020-03-10T18:12:35.050Z	DEBUG	[autodiscover]	cfgfile/list.go:101	Starting runner: elasticsearch (gc, server, slowlog, audit, deprecation)
2020-03-10T18:12:35.050Z	INFO	elasticsearch/client.go:175	Elasticsearch url: https://12c6d839c5e24c469629aaed07bb66d0.us-east-1.aws.found.io:443
2020-03-10T18:12:35.146Z	INFO	elasticsearch/client.go:758	Attempting to connect to Elasticsearch version 7.6.1
2020-03-10T18:12:35.162Z	INFO	[license]	licenser/es_callback.go:50	Elasticsearch license: Platinum
2020-03-10T18:12:35.295Z	INFO	fileset/pipelines.go:134	Elasticsearch pipeline with ID 'filebeat-7.7.0-elasticsearch-server-pipeline' loaded
2020-03-10T18:12:35.419Z	INFO	fileset/pipelines.go:134	Elasticsearch pipeline with ID 'filebeat-7.7.0-elasticsearch-server-pipeline-plaintext' loaded
2020-03-10T18:12:35.536Z	INFO	fileset/pipelines.go:134	Elasticsearch pipeline with ID 'filebeat-7.7.0-elasticsearch-server-pipeline-json' loaded
2020-03-10T18:12:35.652Z	INFO	fileset/pipelines.go:134	Elasticsearch pipeline with ID 'filebeat-7.7.0-elasticsearch-slowlog-pipeline' loaded
2020-03-10T18:12:35.784Z	INFO	fileset/pipelines.go:134	Elasticsearch pipeline with ID 'filebeat-7.7.0-elasticsearch-slowlog-pipeline-plaintext' loaded
2020-03-10T18:12:35.909Z	INFO	fileset/pipelines.go:134	Elasticsearch pipeline with ID 'filebeat-7.7.0-elasticsearch-slowlog-pipeline-json' loaded
2020-03-10T18:12:36.043Z	INFO	fileset/pipelines.go:134	Elasticsearch pipeline with ID 'filebeat-7.7.0-elasticsearch-audit-pipeline' loaded
2020-03-10T18:12:36.186Z	INFO	fileset/pipelines.go:134	Elasticsearch pipeline with ID 'filebeat-7.7.0-elasticsearch-audit-pipeline-json' loaded
2020-03-10T18:12:36.309Z	INFO	fileset/pipelines.go:134	Elasticsearch pipeline with ID 'filebeat-7.7.0-elasticsearch-audit-pipeline-plaintext' loaded
2020-03-10T18:12:36.436Z	INFO	fileset/pipelines.go:134	Elasticsearch pipeline with ID 'filebeat-7.7.0-elasticsearch-deprecation-pipeline' loaded
2020-03-10T18:12:36.595Z	INFO	fileset/pipelines.go:134	Elasticsearch pipeline with ID 'filebeat-7.7.0-elasticsearch-deprecation-pipeline-plaintext' loaded
2020-03-10T18:12:36.720Z	INFO	fileset/pipelines.go:134	Elasticsearch pipeline with ID 'filebeat-7.7.0-elasticsearch-deprecation-pipeline-json' loaded
2020-03-10T18:12:36.860Z	INFO	fileset/pipelines.go:134	Elasticsearch pipeline with ID 'filebeat-7.7.0-elasticsearch-gc-pipeline' loaded
2020-03-10T18:12:36.860Z	INFO	input/input.go:114	Starting input of type: container; ID: 12544767845616830542 
2020-03-10T18:12:36.861Z	INFO	input/input.go:114	Starting input of type: container; ID: 3569229914001296611 
2020-03-10T18:12:36.861Z	INFO	input/input.go:114	Starting input of type: container; ID: 220566605577910699 
2020-03-10T18:12:36.861Z	INFO	input/input.go:114	Starting input of type: container; ID: 16432684558611767041 
2020-03-10T18:12:36.861Z	INFO	input/input.go:114	Starting input of type: container; ID: 8828808671194437362 
2020-03-10T18:12:36.861Z	INFO	log/harvester.go:297	Harvester started for file: /var/log/containers/filebeat-rrxbw_kube-system_filebeat-1bc7e3d9262e847422ace7ef946c39e59abcca679de472207a4af0734a5a859a.log
2020-03-10T18:12:36.862Z	INFO	log/harvester.go:297	Harvester started for file: /var/log/containers/filebeat-rrxbw_kube-system_filebeat-1bc7e3d9262e847422ace7ef946c39e59abcca679de472207a4af0734a5a859a.log
2020-03-10T18:12:36.862Z	INFO	log/harvester.go:297	Harvester started for file: /var/log/containers/filebeat-rrxbw_kube-system_filebeat-1bc7e3d9262e847422ace7ef946c39e59abcca679de472207a4af0734a5a859a.log
2020-03-10T18:12:36.864Z	INFO	log/harvester.go:297	Harvester started for file: /var/log/containers/filebeat-rrxbw_kube-system_filebeat-1bc7e3d9262e847422ace7ef946c39e59abcca679de472207a4af0734a5a859a.log
2020-03-10T18:12:36.864Z	INFO	log/harvester.go:297	Harvester started for file: /var/log/containers/filebeat-rrxbw_kube-system_filebeat-1bc7e3d9262e847422ace7ef946c39e59abcca679de472207a4af0734a5a859a.log
2020-03-10T18:12:37.866Z	INFO	pipeline/output.go:95	Connecting to backoff(elasticsearch(https://12c6d839c5e24c469629aaed07bb66d0.us-east-1.aws.found.io:443))
2020-03-10T18:12:37.924Z	INFO	elasticsearch/client.go:758	Attempting to connect to Elasticsearch version 7.6.1
2020-03-10T18:12:37.941Z	INFO	[license]	licenser/es_callback.go:50	Elasticsearch license: Platinum
2020-03-10T18:12:37.957Z	INFO	[index-management]	idxmgmt/std.go:258	Auto ILM enable success.
2020-03-10T18:12:37.971Z	INFO	[index-management.ilm]	ilm/std.go:139	do not generate ilm policy: exists=true, overwrite=false
2020-03-10T18:12:37.971Z	INFO	[index-management]	idxmgmt/std.go:271	ILM policy successfully loaded.
2020-03-10T18:12:37.971Z	INFO	[index-management]	idxmgmt/std.go:410	Set setup.template.name to '{filebeat-7.7.0 {now/d}-000001}' as ILM is enabled.
2020-03-10T18:12:37.971Z	INFO	[index-management]	idxmgmt/std.go:415	Set setup.template.pattern to 'filebeat-7.7.0-*' as ILM is enabled.
2020-03-10T18:12:37.971Z	INFO	[index-management]	idxmgmt/std.go:449	Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.7.0 {now/d}-000001} as ILM is enabled.
2020-03-10T18:12:37.971Z	INFO	[index-management]	idxmgmt/std.go:453	Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-03-10T18:12:38.177Z	INFO	template/load.go:109	Try loading template filebeat-7.7.0 to Elasticsearch
2020-03-10T18:12:38.442Z	INFO	template/load.go:101	template with name 'filebeat-7.7.0' loaded.
2020-03-10T18:12:38.442Z	INFO	[index-management]	idxmgmt/std.go:295	Loaded index template.
2020-03-10T18:12:38.745Z	INFO	[index-management]	idxmgmt/std.go:306	Write alias successfully generated.

blakerouse · 2020-03-12T13:38:57Z

Attaching log for hints.default_config.enabled: false. I see the same result of it working correctly. Now just to figure out what the difference is.

2020-03-12T13:27:06.233Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:190        Generated config: map[audit:map[enabled:true input:map[paths:[/var/lib/docker/containers/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576/*-json.log] stream:all type:container]] deprecation:map[enabled:true input:map[paths:[/var/lib/docker/containers/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576/*-json.log] stream:all type:container]] gc:map[enabled:true input:map[paths:[/var/lib/docker/containers/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576/*-json.log] stream:all type:container]] module:elasticsearch server:map[enabled:true input:map[paths:[/var/lib/docker/containers/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576/*-json.log] stream:all type:container]] slowlog:map[enabled:true input:map[paths:[/var/lib/docker/containers/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576/*-json.log] stream:all type:container]]]
2020-03-12T13:27:06.233Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:252        Got a meta field in the event
2020-03-12T13:27:06.233Z        DEBUG   [autodiscover]  cfgfile/list.go:62      Starting reload procedure, current runners: 0
2020-03-12T13:27:06.233Z        DEBUG   [autodiscover]  cfgfile/list.go:80      Start list: 1, Stop list: 0
2020-03-12T13:27:06.235Z        DEBUG   [config]        common/config.go:276    load config file '/usr/share/filebeat/module/elasticsearch/audit/manifest.yml' =>
{
  "ingest_pipeline": [
    "ingest/pipeline.json",
    "ingest/pipeline-json.json",
    "ingest/pipeline-plaintext.json"
  ],
  "input": "config/audit.yml",
  "module_version": 1,
  "var": [
    {
      "default": [
        "/var/log/elasticsearch/*_access.log",
        "/var/log/elasticsearch/*_audit.log",
        "/var/log/elasticsearch/*_audit.json"
      ],
      "name": "paths",
      "os": {
        "darwin": [
          "/usr/local/var/lib/elasticsearch/*_access.log",
          "/usr/local/var/lib/elasticsearch/*_audit.log",
          "/usr/local/var/lib/elasticsearch/*_audit.json"
        ],
        "windows": [
          "c:/ProgramData/Elastic/Elasticsearch/logs/*_access.log",
          "c:/ProgramData/Elastic/Elasticsearch/logs/*_audit.log",
          "c:/ProgramData/Elastic/Elasticsearch/logs/*_audit.json"
        ]
      }
    }
  ]
}
2020-03-12T13:27:06.235Z        DEBUG   [config]        common/config.go:276    load config file '/usr/share/filebeat/module/elasticsearch/deprecation/manifest.yml' =>
{
  "ingest_pipeline": [
    "ingest/pipeline.json",
    "ingest/pipeline-plaintext.json",
    "ingest/pipeline-json.json"
  ],
  "input": "config/log.yml",
  "module_version": 1,
  "var": [
    {
      "default": [
        "/var/log/elasticsearch/*_deprecation.log",
        "/var/log/elasticsearch/*_deprecation.json"
      ],
      "name": "paths",
      "os": {
        "darwin": [
          "/usr/local/var/lib/elasticsearch/*_deprecation.log",
          "/usr/local/var/lib/elasticsearch/*_deprecation.json"
        ],
        "windows": [
          "c:/ProgramData/Elastic/Elasticsearch/logs/*_deprecation.log",
          "c:/ProgramData/Elastic/Elasticsearch/logs/*_deprecation.json"
        ]
      }
    }
  ]
}
2020-03-12T13:27:06.236Z        DEBUG   [config]        common/config.go:276    load config file '/usr/share/filebeat/module/elasticsearch/gc/manifest.yml' =>
{
  "ingest_pipeline": "ingest/pipeline.json",
  "input": "config/gc.yml",
  "module_version": 1,
  "var": [
    {
      "default": [
        "/var/log/elasticsearch/gc.log.[0-9]*",
        "/var/log/elasticsearch/gc.log"
      ],
      "name": "paths",
      "os": {
        "darwin": [
          "/usr/local/var/lib/elasticsearch/gc.log.[0-9]*",
          "/usr/local/var/lib/elasticsearch/gc.log"
        ],
        "windows": [
          "c:/ProgramData/Elastic/Elasticsearch/logs/gc.log.*",
          "c:/ProgramData/Elastic/Elasticsearch/logs/gc.log"
        ]
      }
    }
  ]
}
2020-03-12T13:27:06.238Z        DEBUG   [config]        common/config.go:276    load config file '/usr/share/filebeat/module/elasticsearch/server/manifest.yml' =>
{
  "ingest_pipeline": [
    "ingest/pipeline.json",
    "ingest/pipeline-plaintext.json",
    "ingest/pipeline-json.json"
  ],
  "input": "config/log.yml",
  "module_version": 1,
  "var": [
    {
      "default": [
        "/var/log/elasticsearch/*.log",
        "/var/log/elasticsearch/*_server.json"
      ],
      "name": "paths",
      "os": {
        "darwin": [
          "/usr/local/var/lib/elasticsearch/*.log",
          "/usr/local/var/lib/elasticsearch/*_server.json"
        ],
        "windows": [
          "c:/ProgramData/Elastic/Elasticsearch/logs/*.log",
          "c:/ProgramData/Elastic/Elasticsearch/logs/*_server.json"
        ]
      }
    }
  ]
}
2020-03-12T13:27:06.239Z        DEBUG   [config]        common/config.go:276    load config file '/usr/share/filebeat/module/elasticsearch/slowlog/manifest.yml' =>
{
  "ingest_pipeline": [
    "ingest/pipeline.json",
    "ingest/pipeline-plaintext.json",
    "ingest/pipeline-json.json"
  ],
  "input": "config/slowlog.yml",
  "module_version": 1,
  "var": [
    {
      "default": [
        "/var/log/elasticsearch/*_index_search_slowlog.log",
        "/var/log/elasticsearch/*_index_indexing_slowlog.log",
        "/var/log/elasticsearch/*_index_search_slowlog.json",
        "/var/log/elasticsearch/*_index_indexing_slowlog.json"
      ],
      "name": "paths",
      "os": {
        "darwin": [
          "/usr/local/var/lib/elasticsearch/*_index_search_slowlog.log",
          "/usr/local/var/lib/elasticsearch/*_index_indexing_slowlog.log",
          "/usr/local/var/lib/elasticsearch/*_index_search_slowlog.json",
          "/usr/local/var/lib/elasticsearch/*_index_indexing_slowlog.json"
        ],
        "windows": [
          "c:/ProgramData/Elastic/Elasticsearch/logs/*_index_search_slowlog.log",
          "c:/ProgramData/Elastic/Elasticsearch/logs/*_index_indexing_slowlog.log",
          "c:/ProgramData/Elastic/Elasticsearch/logs/*_index_search_slowlog.json",
          "c:/ProgramData/Elastic/Elasticsearch/logs/*_index_indexing_slowlog.json"
        ]
      }
    }
  ]
}
2020-03-12T13:27:06.240Z        DEBUG   [config]        common/config.go:276    Merged input config for fileset elasticsearch/audit
{
  "_fileset_name": "audit",
  "_module_name": "elasticsearch",
  "exclude_files": [
    ".gz$"
  ],
  "path": {
    "config": "/usr/share/filebeat",
    "data": "/usr/share/filebeat/data",
    "home": "/usr/share/filebeat",
    "logs": "/usr/share/filebeat/logs"
  },
  "paths": [
    "/var/lib/docker/containers/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576/*-json.log"
  ],
  "pipeline": "filebeat-7.7.0-elasticsearch-audit-pipeline",
  "processors": [
    {
      "add_locale": null
    }
  ],
  "stream": "all",
  "type": "container"
}
2020-03-12T13:27:06.241Z        DEBUG   [config]        common/config.go:276    Merged input config for fileset elasticsearch/deprecation
{
  "_fileset_name": "deprecation",
  "_module_name": "elasticsearch",
  "exclude_files": [
    ".gz$",
    "_slowlog.log$",
    "_access.log$"
  ],
  "multiline": {
    "match": "after",
    "negate": true,
    "pattern": "^(\\[[0-9]{4}-[0-9]{2}-[0-9]{2}|{)"
  },
  "path": {
    "config": "/usr/share/filebeat",
    "data": "/usr/share/filebeat/data",
    "home": "/usr/share/filebeat",
    "logs": "/usr/share/filebeat/logs"
  },
  "paths": [
    "/var/lib/docker/containers/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576/*-json.log"
  ],
  "pipeline": "filebeat-7.7.0-elasticsearch-deprecation-pipeline",
  "processors": [
    {
      "add_locale": {
        "when": {
          "not": {
            "regexp": {
              "message": "^{"
            }
          }
        }
      }
    }
  ],
  "stream": "all",
  "type": "container"
}
2020-03-12T13:27:06.242Z        DEBUG   [config]        common/config.go:276    Merged input config for fileset elasticsearch/gc
{
  "_fileset_name": "gc",
  "_module_name": "elasticsearch",
  "exclude_files": [
    ".gz$"
  ],
  "exclude_lines": [
    "^(OpenJDK|Java HotSpot).* Server VM ",
    "^CommandLine flags: ",
    "^Memory: "
  ],
  "multiline": {
    "match": "after",
    "negate": true,
    "pattern": "^\\[?[0-9]{4}-[0-9]{2}-[0-9]{2}"
  },
  "path": {
    "config": "/usr/share/filebeat",
    "data": "/usr/share/filebeat/data",
    "home": "/usr/share/filebeat",
    "logs": "/usr/share/filebeat/logs"
  },
  "paths": [
    "/var/lib/docker/containers/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576/*-json.log"
  ],
  "pipeline": "filebeat-7.7.0-elasticsearch-gc-pipeline",
  "stream": "all",
  "type": "container"
}
2020-03-12T13:27:06.243Z        DEBUG   [config]        common/config.go:276    Merged input config for fileset elasticsearch/server
{
  "_fileset_name": "server",
  "_module_name": "elasticsearch",
  "exclude_files": [
    ".gz$",
    "_slowlog.log$",
    "_access.log$",
    "_deprecation.log$"
  ],
  "multiline": {
    "match": "after",
    "negate": true,
    "pattern": "^(\\[[0-9]{4}-[0-9]{2}-[0-9]{2}|{)"
  },
  "path": {
    "config": "/usr/share/filebeat",
    "data": "/usr/share/filebeat/data",
    "home": "/usr/share/filebeat",
    "logs": "/usr/share/filebeat/logs"
  },
  "paths": [
    "/var/lib/docker/containers/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576/*-json.log"
  ],
  "pipeline": "filebeat-7.7.0-elasticsearch-server-pipeline",
  "processors": [
    {
      "add_locale": {
        "when": {
          "not": {
            "regexp": {
              "message": "^{"
            }
          }
        }
      }
    }
  ],
  "stream": "all",
  "type": "container"
}
2020-03-12T13:27:06.244Z        DEBUG   [config]        common/config.go:276    Merged input config for fileset elasticsearch/slowlog
{
  "_fileset_name": "slowlog",
  "_module_name": "elasticsearch",
  "exclude_files": [
    ".gz$"
  ],
  "multiline": {
    "match": "after",
    "negate": true,
    "pattern": "^(\\[?[0-9]{4}-[0-9]{2}-[0-9]{2}|{)"
  },
  "path": {
    "config": "/usr/share/filebeat",
    "data": "/usr/share/filebeat/data",
    "home": "/usr/share/filebeat",
    "logs": "/usr/share/filebeat/logs"
  },
  "paths": [
    "/var/lib/docker/containers/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576/*-json.log"
  ],
  "pipeline": "filebeat-7.7.0-elasticsearch-slowlog-pipeline",
  "processors": [
    {
      "add_locale": {
        "when": {
          "not": {
            "regexp": {
              "message": "^{"
            }
          }
        }
      }
    }
  ],
  "stream": "all",
  "type": "container"
}
2020-03-12T13:27:06.246Z        INFO    log/input.go:152        Configured paths: [/var/lib/docker/containers/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576/*-json.log]
2020-03-12T13:27:06.247Z        DEBUG   [config]        common/config.go:276    Configure processor action 'add_locale' with:
{
  "when": {
    "not": {
      "regexp": {
        "message": "^{"
      }
    }
  }
}
2020-03-12T13:27:06.248Z        INFO    log/input.go:152        Configured paths: [/var/lib/docker/containers/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576/*-json.log]
2020-03-12T13:27:06.249Z        INFO    log/input.go:152        Configured paths: [/var/lib/docker/containers/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576/*-json.log]
2020-03-12T13:27:06.249Z        DEBUG   [config]        common/config.go:276    Configure processor action 'add_locale' with:
{
  "when": {
    "not": {
      "regexp": {
        "message": "^{"
      }
    }
  }
}
2020-03-12T13:27:06.251Z        INFO    log/input.go:152        Configured paths: [/var/lib/docker/containers/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576/*-json.log]
2020-03-12T13:27:06.251Z        DEBUG   [config]        common/config.go:276    Configure processor action 'add_locale' with:
{
  "when": {
    "not": {
      "regexp": {
        "message": "^{"
      }
    }
  }
}
2020-03-12T13:27:06.252Z        INFO    log/input.go:152        Configured paths: [/var/lib/docker/containers/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576/*-json.log]
2020-03-12T13:27:06.252Z        DEBUG   [autodiscover]  cfgfile/list.go:101     Starting runner: elasticsearch (audit, deprecation, gc, server, slowlog)
2020-03-12T13:27:06.252Z        INFO    elasticsearch/client.go:175     Elasticsearch url: https://12c6d839c5e24c469629aaed07bb66d0.us-east-1.aws.found.io:443
2020-03-12T13:27:06.387Z        INFO    elasticsearch/client.go:758     Attempting to connect to Elasticsearch version 7.6.1
2020-03-12T13:27:06.418Z        INFO    [license]       licenser/es_callback.go:50      Elasticsearch license: Platinum
2020-03-12T13:27:06.614Z        INFO    input/input.go:114      Starting input of type: container; ID: 17927908438192879485 
2020-03-12T13:27:06.615Z        INFO    input/input.go:114      Starting input of type: container; ID: 7445362696187540432 
2020-03-12T13:27:06.615Z        INFO    input/input.go:114      Starting input of type: container; ID: 6184799593830925976 
2020-03-12T13:27:06.615Z        INFO    input/input.go:114      Starting input of type: container; ID: 12886999568508837938 
2020-03-12T13:27:06.615Z        INFO    input/input.go:114      Starting input of type: container; ID: 3196839865046264257 
2020-03-12T13:27:06.616Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:165        Got a start event: map[config:[0xc00061a540] host:172.31.16.128 id:1feb963b-6465-11ea-a8a8-02060e2da3b6.elasticsearch kubernetes:{"annotations":{"co":{"elastic":{"logs/enabled":"true","logs/module":"elasticsearch"}},"kubernetes":{"io/psp":"eks.privileged"}},"container":{"id":"674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576","image":"docker.elastic.co/elasticsearch/elasticsearch:7.6.1","name":"elasticsearch","runtime":"docker"},"labels":{"app":"elasticsearch","pod-template-hash":"788b58566"},"namespace":"default","node":{"name":"ip-172-31-21-6.us-east-2.compute.internal"},"pod":{"name":"elasticsearch-788b58566-ksrn9","uid":"1feb963b-6465-11ea-a8a8-02060e2da3b6"},"replicaset":{"name":"elasticsearch-788b58566"}} meta:{"kubernetes":{"container":{"image":"docker.elastic.co/elasticsearch/elasticsearch:7.6.1","name":"elasticsearch"},"labels":{"app":"elasticsearch","pod-template-hash":"788b58566"},"namespace":"default","node":{"name":"ip-172-31-21-6.us-east-2.compute.internal"},"pod":{"name":"elasticsearch-788b58566-ksrn9","uid":"1feb963b-6465-11ea-a8a8-02060e2da3b6"},"replicaset":{"name":"elasticsearch-788b58566"}}} port:9300 provider:e173ebe4-188a-4549-bfcd-0ed57cb9a70d start:true]
2020-03-12T13:27:06.616Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:190        Generated config: map[audit:map[enabled:true input:map[paths:[/var/lib/docker/containers/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576/*-json.log] stream:all type:container]] deprecation:map[enabled:true input:map[paths:[/var/lib/docker/containers/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576/*-json.log] stream:all type:container]] gc:map[enabled:true input:map[paths:[/var/lib/docker/containers/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576/*-json.log] stream:all type:container]] module:elasticsearch server:map[enabled:true input:map[paths:[/var/lib/docker/containers/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576/*-json.log] stream:all type:container]] slowlog:map[enabled:true input:map[paths:[/var/lib/docker/containers/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576/*-json.log] stream:all type:container]]]
2020-03-12T13:27:06.616Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:252        Got a meta field in the event
2020-03-12T13:27:06.616Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:212        Config &{{<nil> } <nil> 0xc0005ad4c0} is already running
2020-03-12T13:27:06.617Z        INFO    log/harvester.go:297    Harvester started for file: /var/lib/docker/containers/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576-json.log
2020-03-12T13:27:06.618Z        INFO    log/harvester.go:297    Harvester started for file: /var/lib/docker/containers/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576-json.log
2020-03-12T13:27:06.619Z        INFO    log/harvester.go:297    Harvester started for file: /var/lib/docker/containers/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576-json.log
2020-03-12T13:27:06.620Z        INFO    log/harvester.go:297    Harvester started for file: /var/lib/docker/containers/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576-json.log
2020-03-12T13:27:06.622Z        INFO    log/harvester.go:297    Harvester started for file: /var/lib/docker/containers/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576/674fe9d1aeba2d76bb005fd52225f984ce96a090e82f3e71005079455d0da576-json.log

blakerouse · 2020-03-12T20:07:39Z

Okay after many hours of debugging and retrying different configurations, I finally broke down and made a change that I thought might be the issue. As it was the only thing that I can see between enabled off and a configuration being provided.

The code looked like this:

	if config, err := from.Child("default_config", -1); err == nil {
		fields := config.GetFields()
		if len(fields) == 1 && fields[0] == "enabled" {
			// only enabling/disabling default config:
			if err := c.DefaultConfig.Merge(config); err != nil {
				return nil
			}
		} else {
			// full config provided, discard default
			c.DefaultConfig = config
		}
	}

Here I could see that every time this was called it was done on a new result of defaultConfig(). So in the case that enabled was the only field under default_config, c.DefaultConfig would be a different object.

But in the case that a more than enabled was provided under default_config, then the config would be taken and set as the default config. But the from.Child("default_config", -1) does not return a copy, its the same child everytime that is passed into Unpack.

So I changed the c.DefaultConfig = config to c.DefaultCount = common.MustNewConfigFrom(config), built a docker image, deployed it, and sure enough it fixed the problem. Being that the default config when defined in the configuration was being shared across multiple inputs it was breaking the expected behavior.

SpencerLN · 2020-04-06T16:12:38Z

According to the release notes for 7.6.2 - it looked like this should be resolved, but we are still seeing this same problem. Can someone confirm that this should indeed be corrected in 7.6.2?

blakerouse · 2020-04-06T21:36:13Z

@SpencerLN This should be fixed in 7.6.2, but GC logs would still have an issue. As you still need #16583 to be resolved, which is not in 7.6.2.

If you only see the issue with GC logs then this issue can be re-closed, but if you are still seeing this issue across many datasets, more digging will need to be done. I was able to test with master at the time with this fix and it did resolve the issue for me.

SpencerLN · 2020-04-06T22:18:58Z

@blakerouse it looks like we are still seeing this for the audit, deprecation, slowlog, and server logs as well when running on Filebeat 7.6.2.

Is there any specific information I can provide to help look into this further?

blakerouse · 2020-04-07T01:07:57Z

@SpencerLN The specific manifest (including filebeat config) you are using to run filebeat along with all annotations on the elasticsearch pod.

SpencerLN · 2020-04-07T14:56:36Z

@blakerouse I attached the filebeat manifest we are using.

These are the labels/annotations on the Elasticsearch pod:

Labels:             common.k8s.elastic.co/type=elasticsearch
                    controller-revision-hash=testcluster-es-data-657bfdbb74
                    elasticsearch.k8s.elastic.co/cluster-name=testcluster
                    elasticsearch.k8s.elastic.co/config-hash=2560483752
                    elasticsearch.k8s.elastic.co/http-scheme=https
                    elasticsearch.k8s.elastic.co/node-data=true
                    elasticsearch.k8s.elastic.co/node-ingest=true
                    elasticsearch.k8s.elastic.co/node-master=false
                    elasticsearch.k8s.elastic.co/node-ml=false
                    elasticsearch.k8s.elastic.co/secure-settings-hash=securesettingshash
                    elasticsearch.k8s.elastic.co/statefulset-name=testcluster-es-data
                    elasticsearch.k8s.elastic.co/version=7.6.2
                    statefulset.kubernetes.io/pod-name=testcluster-es-data-3
Annotations:        co.elastic.logs/enabled: true
                    co.elastic.logs/module: elasticsearch
                    kubernetes.io/limit-ranger:
                      LimitRanger plugin set: cpu request for init container elastic-internal-init-keystore; cpu request for init container install-plugins; cpu...
                    update.k8s.elastic.co/timestamp: 2020-04-01T19:03:13.29170514Z

filebeat-manifest.yaml.txt

botelastic · 2021-04-15T18:16:22Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

SpencerLN · 2021-04-15T18:17:10Z

This is still an ongoing issue.

jb4x · 2022-01-14T16:25:24Z

This was raised in support case 880159 by @SpencerLN, is there an update on this one and when we might be able to expect a fix?

tetianakravchenko · 2022-02-01T09:58:11Z

on version 7.16.2 I could reproduce only duplication of server, slowlog (I suppose deprecation as well, but couldn't reproduce) are duplicated to audit fileset:

SpencerLN added the Filebeat Filebeat label Feb 25, 2020

ChrsMark added autodiscovery Team:Integrations Label for the Integrations team Team:Platforms Label for the Integrations - Platforms team labels Feb 25, 2020

blakerouse added the containers Related to containers use case label Feb 25, 2020

ycombinator mentioned this issue Feb 26, 2020

[Filebeat] Elasticsearch gc fileset collects non-GC logs in container environment #16583

Closed

blakerouse added the [zube]: Investigate label Feb 26, 2020

blakerouse self-assigned this Feb 26, 2020

zube bot unassigned blakerouse Feb 26, 2020

blakerouse self-assigned this Mar 5, 2020

blakerouse added [zube]: In Progress and removed [zube]: Investigate labels Mar 5, 2020

andresrc added [zube]: Inbox [zube]: In Progress and removed [zube]: In Progress [zube]: Inbox labels Mar 9, 2020

blakerouse mentioned this issue Mar 12, 2020

Fix issue where autodiscover hints default configuration was not being copied. #16987

Merged

2 tasks

blakerouse closed this as completed in #16987 Mar 13, 2020

zube bot added [zube]: Done and removed [zube]: In Progress labels Mar 13, 2020

This was referenced Mar 13, 2020

Cherry-pick #16987 to 7.x: Fix issue where autodiscover hints default configuration was not being copied. #17002

Merged

Cherry-pick #16987 to 7.6: Fix issue where autodiscover hints default configuration was not being copied. #17003

Merged

andresrc removed the [zube]: Done label Mar 20, 2020

SpencerLN reopened this Apr 6, 2020

andresrc added the [zube]: Investigate label Apr 7, 2020

blakerouse removed their assignment May 15, 2020

jsoriano mentioned this issue Jan 20, 2021

Elasticsearch server fileset processes logs of other metricsets #23583

Closed

botelastic bot added the Stalled label Apr 15, 2021

botelastic bot removed the Stalled label Apr 15, 2021

masci assigned tetianakravchenko Jan 28, 2022

tetianakravchenko mentioned this issue Feb 2, 2022

Drop not audit logs in elasticsearch/audit fileset ingest pipeline #30164

Merged

6 tasks

tetianakravchenko closed this as completed in #30164 Feb 7, 2022

zube bot added [zube]: Done and removed [zube]: Investigate labels Feb 7, 2022

matschaffer mentioned this issue Feb 17, 2022

Add drop and explicit tests to avoid duplicate ingest of elasticsearch logs #30440

Merged

4 tasks

zube bot removed the [zube]: Done label May 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat] Elasticsearch Module w/ Kubernetes Autodiscover Causes Logs in Incorrect Fieldsets #16540

[Filebeat] Elasticsearch Module w/ Kubernetes Autodiscover Causes Logs in Incorrect Fieldsets #16540

SpencerLN commented Feb 25, 2020

ycombinator commented Feb 25, 2020

blakerouse commented Feb 25, 2020

SpencerLN commented Feb 25, 2020 •

edited

Loading

SpencerLN commented Feb 25, 2020

ycombinator commented Feb 26, 2020

SpencerLN commented Feb 26, 2020 •

edited

Loading

ycombinator commented Feb 26, 2020 •

edited

Loading

blakerouse commented Feb 26, 2020

nerddelphi commented Mar 2, 2020 •

edited

Loading

blakerouse commented Mar 10, 2020

blakerouse commented Mar 12, 2020

blakerouse commented Mar 12, 2020 •

edited

Loading

SpencerLN commented Apr 6, 2020

blakerouse commented Apr 6, 2020

SpencerLN commented Apr 6, 2020

blakerouse commented Apr 7, 2020

SpencerLN commented Apr 7, 2020

botelastic bot commented Apr 15, 2021

SpencerLN commented Apr 15, 2021

jb4x commented Jan 14, 2022

tetianakravchenko commented Feb 1, 2022 •

edited

Loading

[Filebeat] Elasticsearch Module w/ Kubernetes Autodiscover Causes Logs in Incorrect Fieldsets #16540

[Filebeat] Elasticsearch Module w/ Kubernetes Autodiscover Causes Logs in Incorrect Fieldsets #16540

Comments

SpencerLN commented Feb 25, 2020

ycombinator commented Feb 25, 2020

blakerouse commented Feb 25, 2020

SpencerLN commented Feb 25, 2020 • edited Loading

SpencerLN commented Feb 25, 2020

ycombinator commented Feb 26, 2020

SpencerLN commented Feb 26, 2020 • edited Loading

ycombinator commented Feb 26, 2020 • edited Loading

blakerouse commented Feb 26, 2020

nerddelphi commented Mar 2, 2020 • edited Loading

blakerouse commented Mar 10, 2020

blakerouse commented Mar 12, 2020

blakerouse commented Mar 12, 2020 • edited Loading

SpencerLN commented Apr 6, 2020

blakerouse commented Apr 6, 2020

SpencerLN commented Apr 6, 2020

blakerouse commented Apr 7, 2020

SpencerLN commented Apr 7, 2020

botelastic bot commented Apr 15, 2021

SpencerLN commented Apr 15, 2021

jb4x commented Jan 14, 2022

tetianakravchenko commented Feb 1, 2022 • edited Loading

SpencerLN commented Feb 25, 2020 •

edited

Loading

SpencerLN commented Feb 26, 2020 •

edited

Loading

ycombinator commented Feb 26, 2020 •

edited

Loading

nerddelphi commented Mar 2, 2020 •

edited

Loading

blakerouse commented Mar 12, 2020 •

edited

Loading

tetianakravchenko commented Feb 1, 2022 •

edited

Loading