[Fleet]: No data available under Data Streams for installed Agents.

[amolnater-qasource]

Kibana version: 8.4 Snapshot Kibana cloud environment

Host OS and Browser version: All, All

Build details:

VERSION: 8.4.0 Snapshot cloud
BUILD: 53518
COMMIT: 8a54c809495bc08782359073d9392f25c8eb6499

Preconditions:

1. 8.4 Snapshot Kibana cloud environment should be available.
2. Agent should be installed using Policy.

Steps to reproduce:

1. Navigate to Fleet>Agents tab.
2. Observe agents installed Healthy under Agents tab.
3. Navigate to Data Streams and observe no data available for any integrations.

Expected Result:
Data should be available under Data Streams for installed Agents.

Logs:
elastic-agent-diagnostics-2022-06-29T05-32-50Z-00.zip

Screenshots:

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[amolnater-qasource]

@manishgupta-qasource Please review.

[manishgupta-qasource]

Secondary review for this ticket is Done

[jen-huang]

@amolnater-qasource I see you included the agent log files for the windows agent. Could you include them for the Ubuntu agent as well? Going to transfer this to the agent team as this seems to be a data ingestion issue and not a UI one.

[jen-huang]

cc @pierrehilbert

[pierrehilbert]

@amolnater-qasource : As it seems to be the same for each OS, could you please provide me linux agent logs?

[amolnater-qasource]

Hi @jen-huang @pierrehilbert
We have revalidated on latest 8.4 Snapshot Kibana cloud-staging environment and found that the issue is still reproducible.

• No data available under Data Streams for installed Agents.

Build details:
BUILD: 54160
COMMIT: b509d2466e88bf6c4386d8dd5fe89b5c8a54a97f

Logs:
Windows:
Windows.zip

Linux:
Linux.zip

MAC:
MAC.zip

Please let us know if anything else is required from our end.
Thanks

[pierrehilbert]

In addition, could you please share with me the link to the kibana?

[amolnater-qasource]

Hi @pierrehilbert
We have shared the required 8.4 Snapshot Kibana details over slack.

Please let us know if anything else is required from our end.
Thanks

[pierrehilbert]

Hello,
By using the credentials I was able to see an http error:

After that I saw that we have Data Streams in the Stack Management > Index Management > Data Streams

My thoughts are that it's related to Fleet UI. @kpollich will take a look to check if it's the case.

[jen-huang]

Thanks for the investigation @pierrehilbert! Is the lack of event.ingested field something to look into?

Edit: closing loop, Kyle has explanation for missing field here: elastic/kibana#135858

[amolnater-qasource]

Hi @kpollich
We have revalidated this issue on latest 8.4 Snapshot and found it fixed now.

• Data is available under Data Streams for installed Agents.

Build details:
BUILD: 54370
COMMIT: 27befe47a084f7b046426aa3edac01293d6e407b

Screenshot:

Hence marking this as QA:Validated.
Thanks

[amolnater-qasource]

Hi @pierrehilbert
While testing logstash output on 8.4 Snapshot Kibana cloud environment, we have observed this issue reproducible.

Steps followed:

1. Create 3 certs ca, client and logstash with below commands:

elasticsearch-certutil ca --pem
elasticsearch-certutil cert --name logstash --ca-cert C:\elk\elasticsearch\ca\ca.crt --ca-key C:\elk\elasticsearch\ca\ca.key --pem
elasticsearch-certutil cert --name client --ca-cert C:\elk\elasticsearch\ca\ca.crt --ca-key C:\elk\elasticsearch\ca\ca.key --pem

2. Convert the logstash key to pkcs8 as it is the only format supported by logstash:

openssl pkcs8 -inform PEM -in logstash.key -topk8 -nocrypt -outform PEM -out logstash.pkcs8.key

3. Create logstash output-My Logstash Output(could be any):

Add logstash hosts: 10.10.10.10:5044 (machine ip)

Add below certs:

In Server SSL certificate we added- ca certificate.
In Client SSL certificate we added- client certificate.
In Client SSL certificate SSL key we added- client certificate key.

4. Added required configuration in pipelines.yml and created elastic-agent-pipeline.conf.

pipelines.yml:

- pipeline.id: elastic-agent-pipeline
  path.config: "C:\elk\logstash\config\elastic-agent-pipeline.conf"

elastic-agent-pipeline.conf:

input {
  elastic_agent {
    port => 5044
    ssl => true
    ssl_certificate_authorities => ["C:\elk\elasticsearch\ca\ca.crt"]
    ssl_certificate => "C:\elk\elasticsearch\logstash\logstash.crt"
    ssl_key => "C:\elk\elasticsearch\logstash\logstash.pkcs8.key"
    ssl_verify_mode => "force_peer"
  }
}

output {
  elasticsearch {
    hosts => ["https://10.10.10.10:9200"](https://10.10.10.10:9200/)
    api_key => "zorYA4ABQH2oZOWhPX_Q:_Y00OErrQGCRr5tJBE1_zQ"
    data_stream => true
    ssl => true
   # cacert => 
  }
}

5. Run logstash from bin using below command:

logstash -f C:\elk\logstash\config\elastic-agent-pipeline.conf

6. Create new policy with integrations output selected as- My Logstash Output.
7. Run Agent install command for second agent.
8. Observe data for System integration for this installed agent under Data Streams tab.
9. Add OSQuerybeat integration and observe data for OSQuerybeat data too.

After some time Kibana get logged out itself and we observed all data streams removed under Data Streams tab.
However, we have observed new data under Discover tab.

Screenshots:

Build details:

VERSION: 8.4.0
BUILD: 54378
COMMIT: 17a2bcc82856ea7720f684a5f0e2ab0664517401

Hence we are re-opening this issue and sharing the kibana build over slack.

Thanks

[joshdover]

After some time Kibana get logged out itself and we observed all data streams removed under Data Streams tab.
However, we have observed new data under Discover tab.

Want to make sure I understand what's happening here. @amolnater-qasource you were able to see the data streams created from data ingested via Logstash, but then they disappeared after some time even though no ingest changes were made?

Could you check the event.ingested field on the documents you're seeing in discover and share a screenshot of the data including the data_stream.dataset and data_stream.type fields?

[amolnater-qasource]

Hi @joshdover
Thank you for looking into this.

you were able to see the data streams created from data ingested via Logstash, but then they disappeared after some time even though no ingest changes were made?

Yes we were able see the data streams, however after adding both the integrations (System & OSQueryManager) and passing few minutes we observed authentication error and also error pop-ups like shown below:

After few failed attempts of logging in to environment because of session errors and we were able to login to the environment.
However when we navigated to Data Streams tab, none of the previous data streams were available.

Further please find below event.ingested field results under Discover tab:

Json file for elastic_agent.metricbeat dataset:
event-ingested.txt

Please let us know if anything else is required from our end.
Thanks

[joshdover]

Thanks for the information. I think the best course of action would be to have a developer dig into this on the cluster you have (if you haven't deleted it of course). @kpollich would you agree?

[kpollich]

Yes it'd be great to get access to this cluster to take a look at the data streams API response in detail.

[amolnater-qasource]

Hi @kpollich @joshdover
We have shared the existing environment over slack.

Further we will be revalidating the shared issue under #654 (comment) on latest 8.4 Snapshot Kibana environment and will be logging a separate issue if found reproducible again.

Please let us know if anything else is required from our end.
Thanks

[amolnater-qasource]

Hi Team
We have revalidated this issue on latest 8.4 Snapshot and found it reproducible there.

• For reproducing this issue we need to run a live query at OSQuery tab and data under data streams gets removed.

We have logged a separate issue for this at #721
Hence we are closing this.

Thanks