Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stand-alone agent: it's not possible to get agent documents from elasticsearch #20992

Closed
mdelapenya opened this issue Sep 4, 2020 · 8 comments · Fixed by #21047
Closed

Stand-alone agent: it's not possible to get agent documents from elasticsearch #20992

mdelapenya opened this issue Sep 4, 2020 · 8 comments · Fixed by #21047
Assignees
@michalpristas
Labels
bug regression

Comments

@mdelapenya
Copy link
Contributor

  • Version: 8.0.0-SNAPSHOT
  • Operating System:
    • Centos, Dockerised, using the official Stand-Alone image
    • Redhat UBI 8, using the official Stand-Alone image for UBI

This behaviour started to appear on Friday 4th. Before that, it worked as expected.

Steps to Reproduce

  1. Start the elastic stack (ES, Kibana and Package Registry) with docker-compose (I attached a compose file reproducing it: bug.zip
  2. Let the agent work for seconds
  3. Write down the hostname of the container representing the agent
  4. In Kibana, query elasticsearch for indexed documents, using the developer tools (apply $YOUR_HOSTNAME here) and the following query:
GET _search
{
  "version": true,
  "size": 500,
  "docvalue_fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    },
    {
      "field": "system.process.cpu.start_time",
      "format": "date_time"
    },
    {
      "field": "system.service.state_since",
      "format": "date_time"
    }
  ],
  "_source": {
    "excludes": []
  },
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "bool": {
            "filter": [
              {
                "bool": {
                  "should": [
                    {
                      "match_phrase": {
                        "host.name": "${YOUR_HOSTNAME}"
                      }
                    }
                  ],
                  "minimum_should_match": 1
                }
              },
              {
                "bool": {
                  "should": [
                    {
                      "range": {
                        "@timestamp": {
                          "gte": "2020-09-03T14:57:04.948Z"
                        }
                      }
                    }
                  ],
                  "minimum_should_match": 1
                }
              }
            ]
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "2020-09-03T14:57:04.948Z",
              "format": "strict_date_optional_time"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

Search Result:

{
  "took" : 7,
  "timed_out" : false,
  "_shards" : {
    "total" : 13,
    "successful" : 13,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

Agent Log

{"log.level":"info","@timestamp":"2020-09-04T19:45:42.447Z","log.origin":{"file.name":"warn/warn.go","file.line":18},"message":"The Elastic Agent is currently in BETA and should not be used in production","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:42.447Z","log.origin":{"file.name":"application/application.go","file.line":47},"message":"Detecting execution mode","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:42.448Z","log.origin":{"file.name":"application/application.go","file.line":56},"message":"Agent is managed locally","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:42.863Z","log.origin":{"file.name":"application/emitter.go","file.line":171},"message":"Supported programs: Endpoint Security, Filebeat, Heartbeat, Metricbeat","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:42.865Z","log.origin":{"file.name":"application/local_mode.go","file.line":132},"message":"Reloading of configuration is on, frequency is set to 10s","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:42.866Z","log.origin":{"file.name":"application/local_mode.go","file.line":143},"message":"Agent is starting","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:42.867Z","log.origin":{"file.name":"application/local_mode.go","file.line":153},"message":"Agent is stopped","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:42.867Z","log.origin":{"file.name":"application/periodic.go","file.line":60},"message":"Adding 1 file to watch","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:42.867Z","log.origin":{"file.name":"application/periodic.go","file.line":76},"message":"Configuration changes detected","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:42.867Z","log.origin":{"file.name":"application/periodic.go","file.line":82},"message":"Updated 1 files: /usr/share/elastic-agent/elastic-agent.yml","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:42.871Z","log.origin":{"file.name":"application/emitter.go","file.line":145},"message":"Converting single configuration into specific programs configuration","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:42.873Z","log.origin":{"file.name":"application/router.go","file.line":65},"message":"Creating stream: default","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:42.966Z","log.origin":{"file.name":"application/emitter.go","file.line":145},"message":"Converting single configuration into specific programs configuration","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:42.969Z","log.origin":{"file.name":"application/router.go","file.line":65},"message":"Creating stream: default","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:43.632Z","log.origin":{"file.name":"application/router.go","file.line":80},"message":"Streams default need to run config with ID WXdGDNhI and programs: Metricbeat, FLEET_MONITORING","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:43.633Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":47},"message":"New State ID is WXdGDNhI","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:43.633Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":48},"message":"Converging state requires execution of 2 step(s)","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:43.633Z","log.origin":{"file.name":"operation/operator.go","file.line":250},"message":"operator is looking for metricbeat--8.0.0-SNAPSHOT in app collection: map[]","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:43.633Z","log.origin":{"file.name":"operation/operation_fetch.go","file.line":61},"message":"metricbeat.8.0.0-SNAPSHOT already exists in /usr/share/elastic-agent/data/elastic-agent-0032c0/downloads/metricbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz. Skipping operation operation-fetch","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:43.634Z","log.origin":{"file.name":"operation/operator.go","file.line":230},"message":"running operation 'retryable block: operation-fetch operation-verify' for metricbeat.8.0.0-SNAPSHOT","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:43.634Z","log.origin":{"file.name":"operation/operation_fetch.go","file.line":61},"message":"metricbeat.8.0.0-SNAPSHOT already exists in /usr/share/elastic-agent/data/elastic-agent-0032c0/downloads/metricbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz. Skipping operation operation-fetch","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:43.634Z","log.origin":{"file.name":"operation/operation_retryable.go","file.line":83},"message":"running operation 'operation-verify' of the block 'retryable block: operation-fetch operation-verify'","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:43.638Z","log.origin":{"file.name":"application/router.go","file.line":80},"message":"Streams default need to run config with ID WXdGDNhI and programs: Metricbeat, FLEET_MONITORING","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:43.638Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":47},"message":"New State ID is WXdGDNhI","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:43.638Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":48},"message":"Converging state requires execution of 2 step(s)","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:43.638Z","log.origin":{"file.name":"operation/operator.go","file.line":250},"message":"operator is looking for metricbeat--8.0.0-SNAPSHOT in app collection: map[]","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:43.638Z","log.origin":{"file.name":"operation/operation_fetch.go","file.line":61},"message":"metricbeat.8.0.0-SNAPSHOT already exists in /usr/share/elastic-agent/data/elastic-agent-0032c0/downloads/metricbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz. Skipping operation operation-fetch","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:43.638Z","log.origin":{"file.name":"operation/operator.go","file.line":230},"message":"running operation 'retryable block: operation-fetch operation-verify' for metricbeat.8.0.0-SNAPSHOT","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:43.638Z","log.origin":{"file.name":"operation/operation_fetch.go","file.line":61},"message":"metricbeat.8.0.0-SNAPSHOT already exists in /usr/share/elastic-agent/data/elastic-agent-0032c0/downloads/metricbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz. Skipping operation operation-fetch","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:43.638Z","log.origin":{"file.name":"operation/operation_retryable.go","file.line":83},"message":"running operation 'operation-verify' of the block 'retryable block: operation-fetch operation-verify'","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:43.788Z","log.origin":{"file.name":"operation/operator.go","file.line":230},"message":"running operation 'operation-install' for metricbeat.8.0.0-SNAPSHOT","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:43.797Z","log.origin":{"file.name":"operation/operator.go","file.line":226},"message":"operation 'operation-install' skipped for metricbeat.8.0.0-SNAPSHOT","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:43.797Z","log.origin":{"file.name":"operation/operator.go","file.line":230},"message":"running operation 'operation-start' for metricbeat.8.0.0-SNAPSHOT","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:44.012Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2020-09-04T19:45:44Z: type: 'STATE': sub_type: 'STARTING' message: Application: metricbeat--8.0.0-SNAPSHOT[b6d2c1db-fb12-4095-8b1b-9f0d474859c3]: State changed to STARTING: Starting","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:44.013Z","log.origin":{"file.name":"application/periodic.go","file.line":28},"message":"Failed to read configuration, error: could not emit configuration: operator: failed to execute step sc-run, error: failed to start '/usr/share/elastic-agent/data/elastic-agent-0032c0/install/metricbeat-8.0.0-SNAPSHOT-linux-x86_64/metricbeat': fork/exec /usr/share/elastic-agent/data/elastic-agent-0032c0/install/metricbeat-8.0.0-SNAPSHOT-linux-x86_64/metricbeat: text file busy: failed to start '/usr/share/elastic-agent/data/elastic-agent-0032c0/install/metricbeat-8.0.0-SNAPSHOT-linux-x86_64/metricbeat': fork/exec /usr/share/elastic-agent/data/elastic-agent-0032c0/install/metricbeat-8.0.0-SNAPSHOT-linux-x86_64/metricbeat: text file busy","ecs.version":"1.5.0"}
{"log.level":"error","@timestamp":"2020-09-04T19:45:44.013Z","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2020-09-04T19:45:44Z: type: 'ERROR': sub_type: 'FAILED' message: Application: metricbeat--8.0.0-SNAPSHOT[b6d2c1db-fb12-4095-8b1b-9f0d474859c3]: State changed to FAILED: failed to start '/usr/share/elastic-agent/data/elastic-agent-0032c0/install/metricbeat-8.0.0-SNAPSHOT-linux-x86_64/metricbeat': fork/exec /usr/share/elastic-agent/data/elastic-agent-0032c0/install/metricbeat-8.0.0-SNAPSHOT-linux-x86_64/metricbeat: text file busy","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:45.231Z","log.origin":{"file.name":"operation/operator.go","file.line":230},"message":"running operation 'operation-start' for metricbeat.8.0.0-SNAPSHOT","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:45.441Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2020-09-04T19:45:45Z: type: 'STATE': sub_type: 'STARTING' message: Application: metricbeat--8.0.0-SNAPSHOT[b6d2c1db-fb12-4095-8b1b-9f0d474859c3]: State changed to STARTING: Starting","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:45.446Z","log.origin":{"file.name":"operation/operator.go","file.line":230},"message":"running operation 'operation-config' for metricbeat.8.0.0-SNAPSHOT","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:45.451Z","log.origin":{"file.name":"operation/monitoring.go","file.line":256},"message":"monitoring configuration generated for filebeat: map[filebeat:map[inputs:[map[index:logs-elastic.agent-default json:map[keys_under_root:true message_key:message overwrite_keys:true] paths:[/usr/share/elastic-agent/data/elastic-agent-0032c0/logs/elastic-agent-json.log] processors:[map[add_fields:map[fields:map[dataset:elastic.agent namespace:default type:logs] target:data_stream]] map[add_fields:map[fields:map[dataset:elastic.agent] target:event]]] type:log] map[index:logs-elastic.agent.metricbeat-default json:map[keys_under_root:true message_key:message overwrite_keys:true] paths:[/usr/share/elastic-agent/data/elastic-agent-0032c0/logs/default/metricbeat-json.log] processors:[map[add_fields:map[fields:map[dataset:elastic.agent.metricbeat namespace:default type:logs] target:data_stream]] map[add_fields:map[fields:map[dataset:elastic.agent.metricbeat] target:event]]] type:log]]] output:map[elasticsearch:map[hosts:http://elasticsearch:9200 password:changeme type:elasticsearch username:elastic]]]","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:45.451Z","log.origin":{"file.name":"operation/monitoring.go","file.line":305},"message":"monitoring configuration generated for metricbeat: map[metricbeat:map[modules:[map[hosts:[http+unix:///tmp/elastic-agent/default/metricbeat/metricbeat.sock] index:metrics-elastic.agent.metricbeat-default metricsets:[stats state] module:beat period:10s processors:[map[add_fields:map[fields:map[dataset:elastic.agent.metricbeat namespace:default type:metrics] target:data_stream]] map[add_fields:map[fields:map[dataset:elastic.agent.metricbeat] target:event]]]]]] output:map[elasticsearch:map[hosts:http://elasticsearch:9200 password:changeme type:elasticsearch username:elastic]]]","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:45.451Z","log.origin":{"file.name":"operation/operator.go","file.line":250},"message":"operator is looking for filebeat--8.0.0-SNAPSHOT--36643631373035623733363936343635 in app collection: map[metricbeat--8.0.0-SNAPSHOT:0xc0003c37a0]","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:45.451Z","log.origin":{"file.name":"operation/operation_fetch.go","file.line":61},"message":"filebeat.8.0.0-SNAPSHOT already exists in /usr/share/elastic-agent/data/elastic-agent-0032c0/downloads/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz. Skipping operation operation-fetch","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:45.451Z","log.origin":{"file.name":"operation/operator.go","file.line":230},"message":"running operation 'retryable block: operation-fetch operation-verify' for filebeat.8.0.0-SNAPSHOT","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:45.451Z","log.origin":{"file.name":"operation/operation_fetch.go","file.line":61},"message":"filebeat.8.0.0-SNAPSHOT already exists in /usr/share/elastic-agent/data/elastic-agent-0032c0/downloads/filebeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz. Skipping operation operation-fetch","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:45.451Z","log.origin":{"file.name":"operation/operation_retryable.go","file.line":83},"message":"running operation 'operation-verify' of the block 'retryable block: operation-fetch operation-verify'","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:45.611Z","log.origin":{"file.name":"operation/operator.go","file.line":230},"message":"running operation 'operation-install' for filebeat.8.0.0-SNAPSHOT","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:46.681Z","log.origin":{"file.name":"operation/operator.go","file.line":230},"message":"running operation 'operation-start' for filebeat.8.0.0-SNAPSHOT","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:47.073Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2020-09-04T19:45:47Z: type: 'STATE': sub_type: 'STARTING' message: Application: filebeat--8.0.0-SNAPSHOT--36643631373035623733363936343635[b6d2c1db-fb12-4095-8b1b-9f0d474859c3]: State changed to STARTING: Starting","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:47.073Z","log.origin":{"file.name":"operation/operator.go","file.line":230},"message":"running operation 'operation-config' for filebeat.8.0.0-SNAPSHOT","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:47.074Z","log.origin":{"file.name":"operation/operator.go","file.line":250},"message":"operator is looking for metricbeat--8.0.0-SNAPSHOT--36643631373035623733363936343635 in app collection: map[filebeat--8.0.0-SNAPSHOT--36643631373035623733363936343635:0xc00023bc20 metricbeat--8.0.0-SNAPSHOT:0xc0003c37a0]","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:47.074Z","log.origin":{"file.name":"operation/operation_fetch.go","file.line":61},"message":"metricbeat.8.0.0-SNAPSHOT already exists in /usr/share/elastic-agent/data/elastic-agent-0032c0/downloads/metricbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz. Skipping operation operation-fetch","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:47.074Z","log.origin":{"file.name":"operation/operator.go","file.line":230},"message":"running operation 'retryable block: operation-fetch operation-verify' for metricbeat.8.0.0-SNAPSHOT","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:47.074Z","log.origin":{"file.name":"operation/operation_fetch.go","file.line":61},"message":"metricbeat.8.0.0-SNAPSHOT already exists in /usr/share/elastic-agent/data/elastic-agent-0032c0/downloads/metricbeat-8.0.0-SNAPSHOT-linux-x86_64.tar.gz. Skipping operation operation-fetch","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:47.074Z","log.origin":{"file.name":"operation/operation_retryable.go","file.line":83},"message":"running operation 'operation-verify' of the block 'retryable block: operation-fetch operation-verify'","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:47.151Z","log.origin":{"file.name":"operation/operator.go","file.line":226},"message":"operation 'operation-install' skipped for metricbeat.8.0.0-SNAPSHOT","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:47.151Z","log.origin":{"file.name":"operation/operator.go","file.line":230},"message":"running operation 'operation-start' for metricbeat.8.0.0-SNAPSHOT","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:47.405Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2020-09-04T19:45:47Z: type: 'STATE': sub_type: 'STARTING' message: Application: metricbeat--8.0.0-SNAPSHOT--36643631373035623733363936343635[b6d2c1db-fb12-4095-8b1b-9f0d474859c3]: State changed to STARTING: Starting","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:47.405Z","log.origin":{"file.name":"operation/operator.go","file.line":230},"message":"running operation 'operation-config' for metricbeat.8.0.0-SNAPSHOT","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:47.406Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":61},"message":"Updating internal state","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:49.683Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2020-09-04T19:45:49Z: type: 'STATE': sub_type: 'RUNNING' message: Application: metricbeat--8.0.0-SNAPSHOT[b6d2c1db-fb12-4095-8b1b-9f0d474859c3]: State changed to RUNNING: Running","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:51.162Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2020-09-04T19:45:51Z: type: 'STATE': sub_type: 'RUNNING' message: Application: filebeat--8.0.0-SNAPSHOT--36643631373035623733363936343635[b6d2c1db-fb12-4095-8b1b-9f0d474859c3]: State changed to RUNNING: Running","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:51.572Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2020-09-04T19:45:51Z: type: 'STATE': sub_type: 'RUNNING' message: Application: metricbeat--8.0.0-SNAPSHOT--36643631373035623733363936343635[b6d2c1db-fb12-4095-8b1b-9f0d474859c3]: State changed to RUNNING: Running","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:54.014Z","log.origin":{"file.name":"application/periodic.go","file.line":60},"message":"Adding 1 file to watch","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:54.014Z","log.origin":{"file.name":"application/periodic.go","file.line":76},"message":"Configuration changes detected","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:54.014Z","log.origin":{"file.name":"application/periodic.go","file.line":82},"message":"Updated 1 files: /usr/share/elastic-agent/elastic-agent.yml","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:54.017Z","log.origin":{"file.name":"application/emitter.go","file.line":145},"message":"Converting single configuration into specific programs configuration","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:45:54.018Z","log.origin":{"file.name":"application/router.go","file.line":80},"message":"Streams default need to run config with ID WXdGDNhI and programs: Metricbeat, FLEET_MONITORING","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:54.019Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":47},"message":"New State ID is WXdGDNhI","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:54.019Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":48},"message":"Converging state requires execution of 0 step(s)","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:54.019Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":61},"message":"Updating internal state","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:45:54.019Z","log.origin":{"file.name":"application/periodic.go","file.line":98},"message":"No configuration change","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:46:04.020Z","log.origin":{"file.name":"application/periodic.go","file.line":60},"message":"Adding 1 file to watch","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:46:04.020Z","log.origin":{"file.name":"application/periodic.go","file.line":98},"message":"No configuration change","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:46:14.020Z","log.origin":{"file.name":"application/periodic.go","file.line":60},"message":"Adding 1 file to watch","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:46:14.021Z","log.origin":{"file.name":"application/periodic.go","file.line":98},"message":"No configuration change","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:46:24.023Z","log.origin":{"file.name":"application/periodic.go","file.line":60},"message":"Adding 1 file to watch","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:46:24.024Z","log.origin":{"file.name":"application/periodic.go","file.line":98},"message":"No configuration change","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:46:34.024Z","log.origin":{"file.name":"application/periodic.go","file.line":60},"message":"Adding 1 file to watch","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:46:34.024Z","log.origin":{"file.name":"application/periodic.go","file.line":98},"message":"No configuration change","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:46:44.024Z","log.origin":{"file.name":"application/periodic.go","file.line":60},"message":"Adding 1 file to watch","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:46:44.025Z","log.origin":{"file.name":"application/periodic.go","file.line":98},"message":"No configuration change","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:46:54.026Z","log.origin":{"file.name":"application/periodic.go","file.line":60},"message":"Adding 1 file to watch","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:46:54.026Z","log.origin":{"file.name":"application/periodic.go","file.line":98},"message":"No configuration change","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:47:04.026Z","log.origin":{"file.name":"application/periodic.go","file.line":60},"message":"Adding 1 file to watch","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:47:04.026Z","log.origin":{"file.name":"application/periodic.go","file.line":98},"message":"No configuration change","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:47:14.026Z","log.origin":{"file.name":"application/periodic.go","file.line":60},"message":"Adding 1 file to watch","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:47:14.027Z","log.origin":{"file.name":"application/periodic.go","file.line":98},"message":"No configuration change","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:47:24.028Z","log.origin":{"file.name":"application/periodic.go","file.line":60},"message":"Adding 1 file to watch","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:47:24.028Z","log.origin":{"file.name":"application/periodic.go","file.line":98},"message":"No configuration change","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:47:34.029Z","log.origin":{"file.name":"application/periodic.go","file.line":60},"message":"Adding 1 file to watch","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:47:34.029Z","log.origin":{"file.name":"application/periodic.go","file.line":98},"message":"No configuration change","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-09-04T19:47:44.029Z","log.origin":{"file.name":"application/periodic.go","file.line":60},"message":"Adding 1 file to watch","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:47:44.030Z","log.origin":{"file.name":"application/periodic.go","file.line":98},"message":"No configuration change","ecs.version":"1.5.0"}
c{"log.level":"debug","@timestamp":"2020-09-04T19:47:54.032Z","log.origin":{"file.name":"application/periodic.go","file.line":60},"message":"Adding 1 file to watch","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-09-04T19:47:54.032Z","log.origin":{"file.name":"application/periodic.go","file.line":98},"message":"No configuration change","ecs.version":"1.5.0"}

where I can see only one error entry:

{"log.level":"error","@timestamp":"2020-09-04T19:45:44.013Z","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2020-09-04T19:45:44Z: type: 'ERROR': sub_type: 'FAILED' message: Application: metricbeat--8.0.0-SNAPSHOT[b6d2c1db-fb12-4095-8b1b-9f0d474859c3]: State changed to FAILED: failed to start '/usr/share/elastic-agent/data/elastic-agent-0032c0/install/metricbeat-8.0.0-SNAPSHOT-linux-x86_64/metricbeat': fork/exec /usr/share/elastic-agent/data/elastic-agent-0032c0/install/metricbeat-8.0.0-SNAPSHOT-linux-x86_64/metricbeat: text file busy","ecs.version":"1.5.0"}

cc/ @EricDavisX
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ingest-management (Team:Ingest Management)

@mdelapenya mdelapenya changed the title Stand-along agen:t it's not possible to get agent documents from elasticsearch Stand-along agent: it's not possible to get agent documents from elasticsearch Sep 4, 2020
@mdelapenya mdelapenya mentioned this issue Sep 4, 2020
Merged
8 tasks
@mdelapenya mdelapenya changed the title Stand-along agent: it's not possible to get agent documents from elasticsearch Stand-alone agent: it's not possible to get agent documents from elasticsearch Sep 4, 2020
@ph
Copy link
Contributor

ph commented Sep 8, 2020

@michalpristas Can you take a look, it is a regression or the asserts is incorrect?

@EricDavisX
Copy link
Contributor

it would be good to know the state of the Metricbeat / Filebeat processes at that point during the test, it seems likely they are not in proper working order.

It is regression from prior success in the 7.9 and 8.0 branches that test stand-alone mode. I've run the test locally and see the failure. I'll poke into the containers now and report back

@EricDavisX EricDavisX mentioned this issue Sep 8, 2020
Closed
@michalpristas
Copy link
Contributor

This one broke it https://github.com/elastic/beats/pull/20839/files

the thing is that this one creates a new logger which logs just one line
{"log.level":"info","@timestamp":"2020-09-09T15:52:51.939+0200","log.logger":"composable","log.origin":{"file.name":"composable/controller.go","file.line":46},"message":"EXPERIMENTAL - Inputs with variables are currently experimental and should not be used in production","ecs.version":"1.5.0"}

and rotates the one used by main logger. main logger keeps logging into the one which one it thinks is the current one (logger does not see rotation) and therefore filebeat does not pick entries from (from its perspective) stale file (as it gets rotated)

i will fix it next week (PTO tomorrow) or @blakerouse you can pick it up if you have some spare time

@blakerouse
Copy link
Contributor

I guess that logger.New in the context I use it was incorrect. I assumed it still used the same backend.

@michalpristas Is that the issue? Do I need to instead pass the logger in?

@mdelapenya
Copy link
Contributor Author

Meanwhile, we must be aware that the stand-alone tests will be broken until this gets fixed. @EricDavisX how can we effectively communicate this to the team?

@EricDavisX
Copy link
Contributor

Email and slack chat of a quick message is all I can think of for now. Until we get tests in PR CI I think its just you and I that need to know. It does raise the question of communication and 'test skipping' type support as the project matures and we get it into CI that I'm sure will come up.

@michalpristas
Copy link
Contributor

@blakerouse i would suggest passing a logger, and create a named logger if needed

@mdelapenya mdelapenya mentioned this issue Sep 9, 2020
Merged
8 tasks
@blakerouse blakerouse mentioned this issue Sep 9, 2020
Merged
3 tasks
@blakerouse blakerouse mentioned this issue Sep 10, 2020
Merged
3 tasks
@mdelapenya mdelapenya mentioned this issue Sep 14, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@michalpristas michalpristas

Labels
bug regression
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Elastic Agent] Fix loggers in composable module. blakerouse/beats
6 participants
@ph @mdelapenya @michalpristas @blakerouse @EricDavisX @elasticmachine