Logstash monitoring data collection requires Metricbeat to be started after Logstash

[fdartayre]

Logstash version: 7.6.1
Metricbeat version: 7.6.1

Reproduction:

• Start Metricbeat with the logstash-xpack module to collect monitoring data from a Logstash instance
• Start the Logstash instance after Metricbeat

Metricbeat creates the .monitoring-logstash-7-mb- index and indexes documents containing logstash_stats. However the documents don't contain the cluster_uuid field, which causes the Logstash node to not be visible in the monitoring overview in Kibana.

To discover the Logstash nodes, it seems that Kibana sends a query with such a filter:

"filter": [
  {
    "term": {
      "cluster_uuid": {
        "value": "f1zKRFHKQGyfwGgRMOJHDQ",
        "boost": 1
      }
    }
  },
  {
    "range": {
      "logstash_stats.timestamp": {
        "from": null,
        "to": null,
        "include_lower": true,
        "include_upper": true,
        "boost": 1
      }
    }
  }
]

After restarting Metricbeat, subsequent documents contain a cluster_uuid and the Logstash node will appear in monitoring.

[elasticmachine]

Pinging @elastic/stack-monitoring (Stack monitoring)

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[ycombinator]

I can confirm this bug with Metricbeat 7.6.1 as well as 8.0.0 (master).

Note that this only appears to be happening with type: logstash_stats documents, not type: logstash_state documents. So that would narrow it down to the logstash/node_stats metricset code.