[Fleet] Use `unmapped_type: long` and `missing: 0` when sorting datasets that don't include `event.ingested` #136114

kpollich · 2022-07-11T14:47:42Z

Summary

I created a new E2E stack and checked the data streams UI in Fleet before running any tests and saw this error:

{
  "statusCode": 500,
  "error": "Internal Server Error",
  "message": "search_phase_execution_exception: [query_shard_exception] Reason: No mapping found for [event.ingested] in order to sort on"
}

This seemed fixed in elastic/elastic-agent#654 (comment) (see QAS comment), but for some reason this issue is still present in the E2E suite. I've opted to swap to @timestamp which should always exist as a way to unblock E2E tests.

I was able to reproduce this by creating a custom logs integration, ingesting some data, and attempting to load the data streams page in Fleet. Providing an unmapped_type allows Elasticsearch to sort documents even if they don't include event.ingested

elasticmachine · 2022-07-11T14:47:48Z

Pinging @elastic/fleet (Team:Fleet)

kpollich · 2022-07-11T19:00:13Z

@elasticmachine merge upstream

mdelapenya · 2022-07-12T07:37:14Z

I'm currently testing this PR against the e2e tests:

build the docker image for your kibana PR. See https://apm-ci.elastic.co/job/apm-shared/job/oblt-test-env/job/custom-kibana-deploy/12/parameters/
once built (~1h) we go to the E2E framework
make -C .ci create-stack KIBANA_VERSION=pr136114
make -C .ci ssh-stack to SSH into the stack machine, change to the root user (sudo su -) and ~~copy current ".docker/config" to /root/.docker/config.~~ execute docker login -u $YOUR_USER -p $YOUR_PASSWORD docker.elastic.co. This way we will have access to the observability-ci docker registry.
make -C .ci recreate-fleet-server KIBANA_VERSION=pr136114 (yes, it's duplication, but for some reason fleet-server dies because it tries to start too early before kibana. Any help here will be great!!)
make -C .ci list-plataforms to list the OSs and choose one (default is debian 10 amd64)
export NODE_LABEL=debian_10_amd64
make -C .ci create-node
make -C .ci run-tests TAGS="fleet_mode && install"
make -C .ci destroy-stack. Important to destroy the resources!

mdelapenya · 2022-07-12T08:58:21Z

@kpollich with the steps above, I'm seeing the agent never gets online. When I SSH into the machine, and inspect the agent I see:

admin@ip-172-31-32-22:~$ sudo elastic-agent inspect
Error: no fleet config retrieved yet
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.4/fleet-troubleshooting.html
admin@ip-172-31-32-22:~$ sudo elastic-agent status
Status: FAILED
Message: component gateway-3b768786: failed to dispatch actions, error: fail to communicate with updated API client hosts: Get "http://fleet-server:8220/api/status?": lookup fleet-server on 172.31.0.2:53: no such host
Applications: (none)

and TBH I don't know where this http://fleet-server:8220 strings comes from, as we use the stack IP when generating kibana and agent configs. I noticed we are setting the FLEET_URL env var for the agent, has this changed?

The Kibana response is:

time="2022-07-12T08:59:47Z" level=trace msg="Kibana API Query" body=null headers="[]" method=GET url="http://18.218.202.73:5601/api/fleet/agents/26a02818-b0ce-4a8e-b4a9-b3b2a7fa880c"
time="2022-07-12T08:59:47Z" level=trace msg="Agent Status found" agentStatus=error

kpollich · 2022-07-12T12:28:10Z

Hi @mdelapenya thanks for looking into this. I'm continuing to work on this today.

and TBH I don't know where this http://fleet-server:8220 strings comes from, as we use the stack IP when generating kibana and agent configs

The only time I see the fleet-server:8220 string in the Kibana codebase is in reference to Kubernetes setups. We use that value in the K8s manifest file when providing config for deploying agent within k8s.

I noticed we are setting the FLEET_URL env var for the agent, has this changed?

I am not aware of any changes to this environment variable, no.

I'm looking at why Cypress tests are failing on this branch now, but I will try to reproduce the above issue once that's resolved.

kpollich · 2022-07-12T16:32:19Z

I pushed a change in 7e07c5d to filter out non-integrations data streams from the Fleet data streams API. It didn't make sense that we returned all data streams in this API anyway, as Elasticsearch has its own methods for getting that information. This should help with the enterprise search documents ingested under the logs-* and metrics-* data streams causing errors here.

@mdelapenya I am having trouble following your instructions to test here. I get a docker auth error when trying to fetch the docker image for this PR. Can you help out when you're next online? Thanks.

mdelapenya · 2022-07-13T09:08:06Z

@mdelapenya I am having trouble following your instructions to test here. I get a docker auth error when trying to fetch the docker image for this PR. Can you help out when you're next online? Thanks.

Please run docker login in your stack machine (will update the description above), as I verified it's not enough copying the docker config file.

mdelapenya · 2022-07-13T09:42:22Z

@kpollich I'm still seeing this error:

time="2022-07-13T09:36:35Z" level=trace msg="Kibana API Query" body=null headers="[]" method=GET url="http://18.222.197.41:5601/api/fleet/data_streams"
time="2022-07-13T09:36:35Z" level=error msg="Could not get Fleet data streams api" body="{\"statusCode\":500,\"error\":\"Internal Server Error\",\"message\":\"search_phase_execution_exception: [date_time_exception] Reason: Field Year cannot be printed as the value -292275055 exceeds the maximum print width of 4\"}" error="<nil>" statusCode=500

Kibana logs:

[2022-07-13T09:37:41.122+00:00][ERROR][plugins.fleet] ResponseError: search_phase_execution_exception: [date_time_exception] Reason: Field Year cannot be printed as the value -292275055 exceeds the maximum print width of 4
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/src/core/server/elasticsearch/client/create_transport.js:58:16)
    at Client.SearchApi [as search] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/search.js:65:12)
    at async Promise.all (index 0)
    at getDataStreamsQueryMetadata (/usr/share/kibana/x-pack/plugins/fleet/server/routes/data_streams/get_data_streams_query_metadata.js:20:130)
    at /usr/share/kibana/x-pack/plugins/fleet/server/routes/data_streams/handlers.js:117:11
    at async Promise.all (index 8)
    at getListHandler (/usr/share/kibana/x-pack/plugins/fleet/server/routes/data_streams/handlers.js:179:26)
    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:163:30)
    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)
    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)
    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)

API call:

API endpoint: http://18.222.197.41:5601/api/fleet/data_streams
Body: empty
Headers: empty

If I go to the browser and browse the same endpoint, I get the same error message.

Kibana config

---
server.name: kibana
server.host: "0.0.0.0"

telemetry.enabled: false

elasticsearch.hosts: [ "http://18.222.197.41:9200" ]
elasticsearch.username: admin
elasticsearch.password: changeme
xpack.monitoring.ui.container.elasticsearch.enabled: true

xpack.fleet.registryUrl: "https://epr-staging.elastic.co"
xpack.fleet.agents.enabled: true
xpack.fleet.agents.elasticsearch.host: "http://18.222.197.41:9200"
xpack.fleet.agents.fleet_server.hosts: ["http://18.222.197.41:8220"]

xpack.encryptedSavedObjects.encryptionKey: "12345678901234567890123456789012"
xpack.fleet.agents.tlsCheckDisabled: true

xpack.fleet.packages:
  - name: fleet_server
    version: latest
xpack.fleet.agentPolicies:
  - name: Fleet Server policy
    id: fleet-server-policy
    description: Fleet server policy
    namespace: default
    package_policies:
      - name: Fleet Server
        package:
          name: fleet_server

kpollich · 2022-07-13T12:16:28Z

@mdelapenya I'm trying your updated steps above but getting a docker error when trying to spin up kibana in the stack

"Error response from daemon: pull access denied for docker.elastic.co/observability-ci/kibana, repository does not exist or may require 'docker login': denied: requested access to the resource is denied"

This is after running docker login from https://docker-auth.elastic.co/github_auth in both my local shell and in the stack via SSH.

Because Kibana never comes up, the recreate-fleet-server command just errors with a Fleet not ready error.

kpollich · 2022-07-13T16:02:27Z

@mdelapenya Helped me get unblocked on E2E test setup and I discovered something interesting.

In the e2e environment, none of the data streams created seem to contain documents that have been run through Fleet's final-pipeline ingest pipeline that's created during setup. I ran the following query both in the e2e environment and in my local development environment for this branch, with differing results:

GET logs-*,metrics-*/_search
{
  "_source": ["event"]
}

Example records in E2E:

   "hits": [
      {
        "_index": ".ds-logs-elastic_agent-default-2022.07.13-000001",
        "_id": "o5sM-IEBMHzWmRHo_Ii_",
        "_score": 1,
        "_source": {
          "event": {
            "dataset": "elastic_agent"
          }
        }
      },
      {
        "_index": ".ds-logs-elastic_agent-default-2022.07.13-000001",
        "_id": "pJsM-IEBMHzWmRHo_Ii_",
        "_score": 1,
        "_source": {
          "event": {
            "dataset": "elastic_agent"
          }
        }
      },
      {
        "_index": ".ds-logs-elastic_agent-default-2022.07.13-000001",
        "_id": "pZsM-IEBMHzWmRHo_Ii_",
        "_score": 1,
        "_source": {
          "event": {
            "dataset": "elastic_agent"
          }
        }
      },
      {
        "_index": ".ds-logs-elastic_agent-default-2022.07.13-000001",
        "_id": "ppsM-IEBMHzWmRHo_Ii_",
        "_score": 1,
        "_source": {
          "event": {
            "dataset": "elastic_agent"
          }
        }
      },
      {
        "_index": ".ds-logs-elastic_agent-default-2022.07.13-000001",
        "_id": "p5sM-IEBMHzWmRHo_Ii_",
        "_score": 1,
        "_source": {
          "event": {
            "dataset": "elastic_agent"
          }
        }
      },
      {
        "_index": ".ds-logs-elastic_agent-default-2022.07.13-000001",
        "_id": "qJsM-IEBMHzWmRHo_Ii_",
        "_score": 1,
        "_source": {
          "event": {
            "dataset": "elastic_agent"
          }
        }
      },

Example records in local dev environment:

      {
        "_index": ".ds-logs-elastic_agent-default-2022.07.13-000001",
        "_id": "83g5-IEB0QK0PMS9pU9s",
        "_score": 1,
        "_source": {
          "event": {
            "agent_id_status": "verified",
            "ingested": "2022-07-13T15:42:06Z",
            "dataset": "elastic_agent"
          }
        }
      },
      {
        "_index": ".ds-logs-elastic_agent-default-2022.07.13-000001",
        "_id": "9Xg5-IEB0QK0PMS9pU9s",
        "_score": 1,
        "_source": {
          "event": {
            "agent_id_status": "verified",
            "ingested": "2022-07-13T15:42:06Z",
            "dataset": "elastic_agent"
          }
        }
      },
      {
        "_index": ".ds-logs-elastic_agent-default-2022.07.13-000001",
        "_id": "9ng5-IEB0QK0PMS9pU9s",
        "_score": 1,
        "_source": {
          "event": {
            "agent_id_status": "verified",
            "ingested": "2022-07-13T15:42:06Z",
            "dataset": "elastic_agent"
          }
        }
      },
      {
        "_index": ".ds-logs-elastic_agent-default-2022.07.13-000001",
        "_id": "-Hg5-IEB0QK0PMS9pU9s",
        "_score": 1,
        "_source": {
          "event": {
            "agent_id_status": "verified",
            "ingested": "2022-07-13T15:42:06Z",
            "dataset": "elastic_agent"
          }
        }
      },
      {
        "_index": ".ds-logs-elastic_agent-default-2022.07.13-000001",
        "_id": "-Xg5-IEB0QK0PMS9pU9s",
        "_score": 1,
        "_source": {
          "event": {
            "agent_id_status": "verified",
            "ingested": "2022-07-13T15:42:06Z",
            "dataset": "elastic_agent"
          }
        }
      },
      {
        "_index": ".ds-logs-elastic_agent-default-2022.07.13-000001",
        "_id": "-3g5-IEB0QK0PMS9pU9s",
        "_score": 1,
        "_source": {
          "event": {
            "agent_id_status": "verified",
            "ingested": "2022-07-13T15:42:06Z",
            "dataset": "elastic_agent"
          }
        }
      },

Neither the agent_id_status nor the ingested values are present in the records for the E2E tests, suggesting that somehow the .fleet-final-pipeline ingest pipeline isn't being triggered.

I see the expected fleet component templates that set the final_pipeline value to .fleet-final-pipeline-1 as expected, so I don't quite understand how in E2E we could ingest data to these data streams that doesn't go through the final pipeline.

I'm also still not sure where the date value throwing the error Reason: Field Year cannot be printed as the value -292275055 exceeds the maximum print width of 4 is coming from in the E2E tests.

# No hits 
GET logs-*,metrics-*/_search
{
  "query": {
    "multi_match": {
      "query": "-292275055",
      "fields": ["*"]
    }
  }
}

I'm unable to reproduce the specific date parsing error coming from E2E tests in my local development environment, but I will continue investigating.

kpollich · 2022-07-13T16:08:26Z

Here's a comparison of two roughly equivalent records - E2E on the left, my local self hosted dev environment on the right:

It makes sense that cloud is missing from the self hosted agent, so no issues there. But you can see the event values differ.

kpollich · 2022-07-13T16:21:23Z

Building a new docker image off of ce09f35 where I switched to using @timestamp to sort instead of event.ingested. I don't why or how these documents are bypassing the fleet final pipeline, but I think it's safe to use @timestamp instead to hopefully unblock.

kpollich · 2022-07-13T18:11:07Z

So turns out my assumption that we can use @timestamp instead of event.ingested here is incorrect. Back to the drawing board.

mdelapenya · 2022-07-14T07:12:28Z

@kpollich just to add more context, this error happens only for main (8.4), not in 8.3, 8.2 or 7.17

kpollich · 2022-07-14T14:28:23Z

@mdelapenya I was finally able to reproduce this by setting up a custom logs integration (which doesn't include event.ingested - see updated description), so I'm hopeful that I can get a working E2E run done once this PR is green -> docker image is built.

kpollich · 2022-07-14T14:28:58Z

FWIW it's still a concern to me that the documents we ingest during E2E tests seemingly don't go through the .fleet-final-pipeline ingest pipeline.

mdelapenya · 2022-07-14T14:33:10Z

FWIW it's still a concern to me that the documents we ingest during E2E tests seemingly don't go through the .fleet-final-pipeline ingest pipeline.

In that sense, we can pair and follow the code in the E2E, but they basically invoke the install command in the elastic-agent. Is it possible that fleet-server needs to be configured/updated in any other manner?

kpollich · 2022-07-14T17:31:43Z

~~@mdelapenya Seems like the test is passing and the data streams page/API are working properly against the latest docker image for this PR :)~~

Ugh maybe not. A second run of the test resulted in errors. Still looking :/

kpollich · 2022-07-14T17:32:12Z

@nchaulet I re-requested review here since the actual scope of changes has been altered. Title/description updated to match.

kpollich · 2022-07-14T18:00:45Z

I managed to shell into the docker container running Kibana in my E2E stack and isolate the print width error to a single data stream. Here's the exact query we run in the data streams API now and its results on this data stream:

GET metrics-elastic_agent.fleet_server-default/_search
{
  "_source": false, 
  "fields": [
    "event.ingested"
  ], 
  "sort": {
    "event.ingested": {
      "order": "desc",
      "unmapped_type": "long"
    }
  }
}

{
  "error": {
    "root_cause": [
      {
        "type": "date_time_exception",
        "reason": "Field Year cannot be printed as the value -292275055 exceeds the maximum print width of 4"
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": ".ds-metrics-elastic_agent.fleet_server-default-2022.07.14-000001",
        "node": "gQ1xgYorSC2z1JfrajVxig",
        "reason": {
          "type": "date_time_exception",
          "reason": "Field Year cannot be printed as the value -292275055 exceeds the maximum print width of 4"
        }
      }
    ],
    "caused_by": {
      "type": "date_time_exception",
      "reason": "Field Year cannot be printed as the value -292275055 exceeds the maximum print width of 4",
      "caused_by": {
        "type": "date_time_exception",
        "reason": "Field Year cannot be printed as the value -292275055 exceeds the maximum print width of 4"
      }
    }
  },
  "status": 500
}

Here's all the documents under that data stream that I see in my E2E cluster

GET metrics-elastic_agent.fleet_server-default/_search 
{
}

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 218,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".ds-metrics-elastic_agent.fleet_server-default-2022.07.14-000001",
        "_id": "7ZPd_YEB2E_yqH41HmNX",
        "_score": 1,
        "_source": {
          "cloud": {
            "availability_zone": "us-east-2c",
            "image": {
              "id": "ami-0d90bed76900e679a"
            },
            "instance": {
              "id": "i-03d62e996578c5dc5"
            },
            "provider": "aws",
            "service": {
              "name": "EC2"
            },
            "machine": {
              "type": "t3.xlarge"
            },
            "region": "us-east-2",
            "account": {
              "id": "946960629917"
            }
          },
          "agent": {
            "name": "8687b5db5dcb",
            "id": "3c00a3e3-b922-440b-9176-997cb431777b",
            "ephemeral_id": "7bf61f4b-3387-40e0-913c-11483ce70443",
            "type": "metricbeat",
            "version": "8.4.0"
          },
          "@timestamp": "2022-07-14T17:58:54.826Z",
          "ecs": {
            "version": "8.0.0"
          },
          "service": {
            "address": "http://unix/state",
            "type": "beat"
          },
          "data_stream": {
            "namespace": "default",
            "type": "metrics",
            "dataset": "elastic_agent.fleet_server"
          },
          "beat": {
            "state": {
              "cluster": {
                "uuid": ""
              },
              "beat": {
                "name": "fleet-server",
                "host": "",
                "type": "",
                "version": "8.4.0-SNAPSHOT",
                "uuid": ""
              }
            }
          },
          "elastic_agent": {
            "id": "3c00a3e3-b922-440b-9176-997cb431777b",
            "version": "8.4.0",
            "snapshot": true
          },
          "host": {
            "hostname": "8687b5db5dcb",
            "os": {
              "kernel": "4.19.0-18-cloud-amd64",
              "codename": "focal",
              "name": "Ubuntu",
              "type": "linux",
              "family": "debian",
              "version": "20.04.4 LTS (Focal Fossa)",
              "platform": "ubuntu"
            },
            "containerized": true,
            "ip": [
              "172.18.0.4"
            ],
            "name": "8687b5db5dcb",
            "mac": [
              "02:42:ac:12:00:04"
            ],
            "architecture": "x86_64"
          },
          "metricset": {
            "period": 10000,
            "name": "state"
          },
          "event": {
            "duration": 2421885,
            "module": "beat",
            "dataset": "elastic_agent.fleet_server"
          }
        }
      },
      {
        "_index": ".ds-metrics-elastic_agent.fleet_server-default-2022.07.14-000001",
        "_id": "65Pc_YEB2E_yqH4192NI",
        "_score": 1,
        "_source": {
          "cloud": {
            "availability_zone": "us-east-2c",
            "image": {
              "id": "ami-0d90bed76900e679a"
            },
            "instance": {
              "id": "i-03d62e996578c5dc5"
            },
            "provider": "aws",
            "machine": {
              "type": "t3.xlarge"
            },
            "service": {
              "name": "EC2"
            },
            "region": "us-east-2",
            "account": {
              "id": "946960629917"
            }
          },
          "agent": {
            "name": "8687b5db5dcb",
            "id": "3c00a3e3-b922-440b-9176-997cb431777b",
            "ephemeral_id": "7bf61f4b-3387-40e0-913c-11483ce70443",
            "type": "metricbeat",
            "version": "8.4.0"
          },
          "@timestamp": "2022-07-14T17:58:44.827Z",
          "ecs": {
            "version": "8.0.0"
          },
          "service": {
            "address": "http://unix/state",
            "type": "beat"
          },
          "data_stream": {
            "namespace": "default",
            "type": "metrics",
            "dataset": "elastic_agent.fleet_server"
          },
          "beat": {
            "state": {
              "cluster": {
                "uuid": ""
              },
              "beat": {
                "name": "fleet-server",
                "host": "",
                "type": "",
                "uuid": "",
                "version": "8.4.0-SNAPSHOT"
              }
            }
          },
          "elastic_agent": {
            "id": "3c00a3e3-b922-440b-9176-997cb431777b",
            "version": "8.4.0",
            "snapshot": true
          },
          "host": {
            "hostname": "8687b5db5dcb",
            "os": {
              "kernel": "4.19.0-18-cloud-amd64",
              "codename": "focal",
              "name": "Ubuntu",
              "type": "linux",
              "family": "debian",
              "version": "20.04.4 LTS (Focal Fossa)",
              "platform": "ubuntu"
            },
            "containerized": true,
            "ip": [
              "172.18.0.4"
            ],
            "name": "8687b5db5dcb",
            "mac": [
              "02:42:ac:12:00:04"
            ],
            "architecture": "x86_64"
          },
          "metricset": {
            "period": 10000,
            "name": "state"
          },
          "event": {
            "duration": 1861869,
            "module": "beat",
            "dataset": "elastic_agent.fleet_server"
          }
        }
      },
      {
        "_index": ".ds-metrics-elastic_agent.fleet_server-default-2022.07.14-000001",
        "_id": "55Pc_YEB2E_yqH410GM3",
        "_score": 1,
        "_source": {
          "cloud": {
            "availability_zone": "us-east-2c",
            "image": {
              "id": "ami-0d90bed76900e679a"
            },
            "instance": {
              "id": "i-03d62e996578c5dc5"
            },
            "provider": "aws",
            "service": {
              "name": "EC2"
            },
            "machine": {
              "type": "t3.xlarge"
            },
            "region": "us-east-2",
            "account": {
              "id": "946960629917"
            }
          },
          "agent": {
            "name": "8687b5db5dcb",
            "id": "3c00a3e3-b922-440b-9176-997cb431777b",
            "ephemeral_id": "7bf61f4b-3387-40e0-913c-11483ce70443",
            "type": "metricbeat",
            "version": "8.4.0"
          },
          "@timestamp": "2022-07-14T17:58:34.827Z",
          "ecs": {
            "version": "8.0.0"
          },
          "data_stream": {
            "namespace": "default",
            "type": "metrics",
            "dataset": "elastic_agent.fleet_server"
          },
          "service": {
            "address": "http://unix/state",
            "type": "beat"
          },
          "host": {
            "hostname": "8687b5db5dcb",
            "os": {
              "kernel": "4.19.0-18-cloud-amd64",
              "codename": "focal",
              "name": "Ubuntu",
              "type": "linux",
              "family": "debian",
              "version": "20.04.4 LTS (Focal Fossa)",
              "platform": "ubuntu"
            },
            "containerized": true,
            "ip": [
              "172.18.0.4"
            ],
            "name": "8687b5db5dcb",
            "mac": [
              "02:42:ac:12:00:04"
            ],
            "architecture": "x86_64"
          },
          "beat": {
            "state": {
              "cluster": {
                "uuid": ""
              },
              "beat": {
                "name": "fleet-server",
                "host": "",
                "type": "",
                "uuid": "",
                "version": "8.4.0-SNAPSHOT"
              }
            }
          },
          "elastic_agent": {
            "id": "3c00a3e3-b922-440b-9176-997cb431777b",
            "version": "8.4.0",
            "snapshot": true
          },
          "metricset": {
            "period": 10000,
            "name": "state"
          },
          "event": {
            "duration": 621623,
            "module": "beat",
            "dataset": "elastic_agent.fleet_server"
          }
        }
      },
      {
        "_index": ".ds-metrics-elastic_agent.fleet_server-default-2022.07.14-000001",
        "_id": "5JPc_YEB2E_yqH41qWMm",
        "_score": 1,
        "_source": {
          "cloud": {
            "image": {
              "id": "ami-0d90bed76900e679a"
            },
            "availability_zone": "us-east-2c",
            "instance": {
              "id": "i-03d62e996578c5dc5"
            },
            "provider": "aws",
            "machine": {
              "type": "t3.xlarge"
            },
            "service": {
              "name": "EC2"
            },
            "region": "us-east-2",
            "account": {
              "id": "946960629917"
            }
          },
          "agent": {
            "name": "8687b5db5dcb",
            "id": "3c00a3e3-b922-440b-9176-997cb431777b",
            "type": "metricbeat",
            "ephemeral_id": "7bf61f4b-3387-40e0-913c-11483ce70443",
            "version": "8.4.0"
          },
          "@timestamp": "2022-07-14T17:58:24.826Z",
          "ecs": {
            "version": "8.0.0"
          },
          "service": {
            "address": "http://unix/state",
            "type": "beat"
          },
          "data_stream": {
            "namespace": "default",
            "type": "metrics",
            "dataset": "elastic_agent.fleet_server"
          },
          "beat": {
            "state": {
              "cluster": {
                "uuid": ""
              },
              "beat": {
                "host": "",
                "name": "fleet-server",
                "type": "",
                "uuid": "",
                "version": "8.4.0-SNAPSHOT"
              }
            }
          },
          "elastic_agent": {
            "id": "3c00a3e3-b922-440b-9176-997cb431777b",
            "version": "8.4.0",
            "snapshot": true
          },
          "host": {
            "hostname": "8687b5db5dcb",
            "os": {
              "kernel": "4.19.0-18-cloud-amd64",
              "codename": "focal",
              "name": "Ubuntu",
              "type": "linux",
              "family": "debian",
              "version": "20.04.4 LTS (Focal Fossa)",
              "platform": "ubuntu"
            },
            "containerized": true,
            "ip": [
              "172.18.0.4"
            ],
            "name": "8687b5db5dcb",
            "mac": [
              "02:42:ac:12:00:04"
            ],
            "architecture": "x86_64"
          },
          "metricset": {
            "period": 10000,
            "name": "state"
          },
          "event": {
            "duration": 853140,
            "module": "beat",
            "dataset": "elastic_agent.fleet_server"
          }
        }
      },
      {
        "_index": ".ds-metrics-elastic_agent.fleet_server-default-2022.07.14-000001",
        "_id": "4ZPc_YEB2E_yqH41gmMW",
        "_score": 1,
        "_source": {
          "cloud": {
            "availability_zone": "us-east-2c",
            "image": {
              "id": "ami-0d90bed76900e679a"
            },
            "instance": {
              "id": "i-03d62e996578c5dc5"
            },
            "provider": "aws",
            "service": {
              "name": "EC2"
            },
            "machine": {
              "type": "t3.xlarge"
            },
            "region": "us-east-2",
            "account": {
              "id": "946960629917"
            }
          },
          "agent": {
            "name": "8687b5db5dcb",
            "id": "3c00a3e3-b922-440b-9176-997cb431777b",
            "ephemeral_id": "7bf61f4b-3387-40e0-913c-11483ce70443",
            "type": "metricbeat",
            "version": "8.4.0"
          },
          "@timestamp": "2022-07-14T17:58:14.826Z",
          "ecs": {
            "version": "8.0.0"
          },
          "service": {
            "address": "http://unix/state",
            "type": "beat"
          },
          "data_stream": {
            "namespace": "default",
            "type": "metrics",
            "dataset": "elastic_agent.fleet_server"
          },
          "host": {
            "hostname": "8687b5db5dcb",
            "os": {
              "kernel": "4.19.0-18-cloud-amd64",
              "codename": "focal",
              "name": "Ubuntu",
              "type": "linux",
              "family": "debian",
              "version": "20.04.4 LTS (Focal Fossa)",
              "platform": "ubuntu"
            },
            "containerized": true,
            "ip": [
              "172.18.0.4"
            ],
            "name": "8687b5db5dcb",
            "mac": [
              "02:42:ac:12:00:04"
            ],
            "architecture": "x86_64"
          },
          "elastic_agent": {
            "id": "3c00a3e3-b922-440b-9176-997cb431777b",
            "version": "8.4.0",
            "snapshot": true
          },
          "beat": {
            "state": {
              "cluster": {
                "uuid": ""
              },
              "beat": {
                "name": "fleet-server",
                "host": "",
                "type": "",
                "uuid": "",
                "version": "8.4.0-SNAPSHOT"
              }
            }
          },
          "metricset": {
            "period": 10000,
            "name": "state"
          },
          "event": {
            "duration": 528866,
            "module": "beat",
            "dataset": "elastic_agent.fleet_server"
          }
        }
      },
      {
        "_index": ".ds-metrics-elastic_agent.fleet_server-default-2022.07.14-000001",
        "_id": "3pPc_YEB2E_yqH41W2MH",
        "_score": 1,
        "_source": {
          "cloud": {
            "availability_zone": "us-east-2c",
            "image": {
              "id": "ami-0d90bed76900e679a"
            },
            "instance": {
              "id": "i-03d62e996578c5dc5"
            },
            "provider": "aws",
            "machine": {
              "type": "t3.xlarge"
            },
            "service": {
              "name": "EC2"
            },
            "region": "us-east-2",
            "account": {
              "id": "946960629917"
            }
          },
          "agent": {
            "name": "8687b5db5dcb",
            "id": "3c00a3e3-b922-440b-9176-997cb431777b",
            "ephemeral_id": "7bf61f4b-3387-40e0-913c-11483ce70443",
            "type": "metricbeat",
            "version": "8.4.0"
          },
          "@timestamp": "2022-07-14T17:58:04.827Z",
          "ecs": {
            "version": "8.0.0"
          },
          "data_stream": {
            "namespace": "default",
            "type": "metrics",
            "dataset": "elastic_agent.fleet_server"
          },
          "service": {
            "address": "http://unix/state",
            "type": "beat"
          },
          "elastic_agent": {
            "id": "3c00a3e3-b922-440b-9176-997cb431777b",
            "version": "8.4.0",
            "snapshot": true
          },
          "host": {
            "hostname": "8687b5db5dcb",
            "os": {
              "kernel": "4.19.0-18-cloud-amd64",
              "codename": "focal",
              "name": "Ubuntu",
              "type": "linux",
              "family": "debian",
              "version": "20.04.4 LTS (Focal Fossa)",
              "platform": "ubuntu"
            },
            "containerized": true,
            "ip": [
              "172.18.0.4"
            ],
            "name": "8687b5db5dcb",
            "mac": [
              "02:42:ac:12:00:04"
            ],
            "architecture": "x86_64"
          },
          "beat": {
            "state": {
              "cluster": {
                "uuid": ""
              },
              "beat": {
                "name": "fleet-server",
                "host": "",
                "type": "",
                "uuid": "",
                "version": "8.4.0-SNAPSHOT"
              }
            }
          },
          "metricset": {
            "period": 10000,
            "name": "state"
          },
          "event": {
            "duration": 777401,
            "module": "beat",
            "dataset": "elastic_agent.fleet_server"
          }
        }
      },
      {
        "_index": ".ds-metrics-elastic_agent.fleet_server-default-2022.07.14-000001",
        "_id": "LJO8_YEB2E_yqH41TldV",
        "_score": 1,
        "_source": {
          "@timestamp": "2022-07-14T17:23:04.425Z",
          "agent": {
            "id": "e41e0a33-7b6b-45fa-9f8e-c66062a45b06",
            "version": "8.4.0",
            "ephemeral_id": "54ad5c8b-3495-4939-bee5-685c77676dd5",
            "name": "9edb93fcacd9",
            "type": "metricbeat"
          },
          "ecs": {
            "version": "8.0.0"
          },
          "beat": {
            "state": {
              "beat": {
                "version": "8.4.0-SNAPSHOT",
                "name": "fleet-server",
                "host": "",
                "type": "",
                "uuid": ""
              },
              "cluster": {
                "uuid": ""
              }
            }
          },
          "metricset": {
            "name": "state",
            "period": 10000
          },
          "data_stream": {
            "type": "metrics",
            "dataset": "elastic_agent.fleet_server",
            "namespace": "default"
          },
          "elastic_agent": {
            "id": "e41e0a33-7b6b-45fa-9f8e-c66062a45b06",
            "snapshot": true,
            "version": "8.4.0"
          },
          "service": {
            "address": "http://unix/state",
            "type": "beat"
          },
          "event": {
            "dataset": "elastic_agent.fleet_server",
            "module": "beat",
            "duration": 1236761
          },
          "host": {
            "name": "9edb93fcacd9",
            "containerized": true,
            "ip": [
              "172.18.0.4"
            ],
            "mac": [
              "02:42:ac:12:00:04"
            ],
            "hostname": "9edb93fcacd9",
            "architecture": "x86_64",
            "os": {
              "family": "debian",
              "name": "Ubuntu",
              "kernel": "4.19.0-18-cloud-amd64",
              "codename": "focal",
              "type": "linux",
              "platform": "ubuntu",
              "version": "20.04.4 LTS (Focal Fossa)"
            }
          },
          "cloud": {
            "service": {
              "name": "EC2"
            },
            "account": {
              "id": "946960629917"
            },
            "image": {
              "id": "ami-0d90bed76900e679a"
            },
            "instance": {
              "id": "i-03d62e996578c5dc5"
            },
            "machine": {
              "type": "t3.xlarge"
            },
            "region": "us-east-2",
            "availability_zone": "us-east-2c",
            "provider": "aws"
          }
        }
      },
      {
        "_index": ".ds-metrics-elastic_agent.fleet_server-default-2022.07.14-000001",
        "_id": "L5O8_YEB2E_yqH41dVdk",
        "_score": 1,
        "_source": {
          "@timestamp": "2022-07-14T17:23:14.425Z",
          "service": {
            "address": "http://unix/state",
            "type": "beat"
          },
          "beat": {
            "state": {
              "cluster": {
                "uuid": ""
              },
              "beat": {
                "host": "",
                "type": "",
                "uuid": "",
                "version": "8.4.0-SNAPSHOT",
                "name": "fleet-server"
              }
            }
          },
          "agent": {
            "ephemeral_id": "54ad5c8b-3495-4939-bee5-685c77676dd5",
            "name": "9edb93fcacd9",
            "type": "metricbeat",
            "id": "e41e0a33-7b6b-45fa-9f8e-c66062a45b06",
            "version": "8.4.0"
          },
          "ecs": {
            "version": "8.0.0"
          },
          "cloud": {
            "image": {
              "id": "ami-0d90bed76900e679a"
            },
            "instance": {
              "id": "i-03d62e996578c5dc5"
            },
            "machine": {
              "type": "t3.xlarge"
            },
            "region": "us-east-2",
            "availability_zone": "us-east-2c",
            "provider": "aws",
            "service": {
              "name": "EC2"
            },
            "account": {
              "id": "946960629917"
            }
          },
          "event": {
            "module": "beat",
            "duration": 788570,
            "dataset": "elastic_agent.fleet_server"
          },
          "metricset": {
            "name": "state",
            "period": 10000
          },
          "data_stream": {
            "namespace": "default",
            "type": "metrics",
            "dataset": "elastic_agent.fleet_server"
          },
          "elastic_agent": {
            "version": "8.4.0",
            "id": "e41e0a33-7b6b-45fa-9f8e-c66062a45b06",
            "snapshot": true
          },
          "host": {
            "containerized": true,
            "ip": [
              "172.18.0.4"
            ],
            "mac": [
              "02:42:ac:12:00:04"
            ],
            "hostname": "9edb93fcacd9",
            "name": "9edb93fcacd9",
            "architecture": "x86_64",
            "os": {
              "family": "debian",
              "name": "Ubuntu",
              "kernel": "4.19.0-18-cloud-amd64",
              "codename": "focal",
              "type": "linux",
              "platform": "ubuntu",
              "version": "20.04.4 LTS (Focal Fossa)"
            }
          }
        }
      },
      {
        "_index": ".ds-metrics-elastic_agent.fleet_server-default-2022.07.14-000001",
        "_id": "MpO8_YEB2E_yqH41nFd1",
        "_score": 1,
        "_source": {
          "@timestamp": "2022-07-14T17:23:24.426Z",
          "metricset": {
            "period": 10000,
            "name": "state"
          },
          "elastic_agent": {
            "version": "8.4.0",
            "id": "e41e0a33-7b6b-45fa-9f8e-c66062a45b06",
            "snapshot": true
          },
          "agent": {
            "id": "e41e0a33-7b6b-45fa-9f8e-c66062a45b06",
            "version": "8.4.0",
            "ephemeral_id": "54ad5c8b-3495-4939-bee5-685c77676dd5",
            "name": "9edb93fcacd9",
            "type": "metricbeat"
          },
          "host": {
            "os": {
              "version": "20.04.4 LTS (Focal Fossa)",
              "family": "debian",
              "name": "Ubuntu",
              "kernel": "4.19.0-18-cloud-amd64",
              "codename": "focal",
              "type": "linux",
              "platform": "ubuntu"
            },
            "containerized": true,
            "ip": [
              "172.18.0.4"
            ],
            "name": "9edb93fcacd9",
            "mac": [
              "02:42:ac:12:00:04"
            ],
            "hostname": "9edb93fcacd9",
            "architecture": "x86_64"
          },
          "cloud": {
            "provider": "aws",
            "service": {
              "name": "EC2"
            },
            "account": {
              "id": "946960629917"
            },
            "image": {
              "id": "ami-0d90bed76900e679a"
            },
            "instance": {
              "id": "i-03d62e996578c5dc5"
            },
            "machine": {
              "type": "t3.xlarge"
            },
            "region": "us-east-2",
            "availability_zone": "us-east-2c"
          },
          "service": {
            "address": "http://unix/state",
            "type": "beat"
          },
          "event": {
            "dataset": "elastic_agent.fleet_server",
            "module": "beat",
            "duration": 723680
          },
          "beat": {
            "state": {
              "cluster": {
                "uuid": ""
              },
              "beat": {
                "version": "8.4.0-SNAPSHOT",
                "name": "fleet-server",
                "host": "",
                "type": "",
                "uuid": ""
              }
            }
          },
          "data_stream": {
            "namespace": "default",
            "type": "metrics",
            "dataset": "elastic_agent.fleet_server"
          },
          "ecs": {
            "version": "8.0.0"
          }
        }
      },
      {
        "_index": ".ds-metrics-elastic_agent.fleet_server-default-2022.07.14-000001",
        "_id": "NZO8_YEB2E_yqH41w1eE",
        "_score": 1,
        "_source": {
          "@timestamp": "2022-07-14T17:23:34.425Z",
          "elastic_agent": {
            "version": "8.4.0",
            "id": "e41e0a33-7b6b-45fa-9f8e-c66062a45b06",
            "snapshot": true
          },
          "agent": {
            "type": "metricbeat",
            "version": "8.4.0",
            "ephemeral_id": "54ad5c8b-3495-4939-bee5-685c77676dd5",
            "id": "e41e0a33-7b6b-45fa-9f8e-c66062a45b06",
            "name": "9edb93fcacd9"
          },
          "ecs": {
            "version": "8.0.0"
          },
          "metricset": {
            "name": "state",
            "period": 10000
          },
          "data_stream": {
            "dataset": "elastic_agent.fleet_server",
            "namespace": "default",
            "type": "metrics"
          },
          "event": {
            "dataset": "elastic_agent.fleet_server",
            "module": "beat",
            "duration": 853584
          },
          "host": {
            "containerized": true,
            "ip": [
              "172.18.0.4"
            ],
            "mac": [
              "02:42:ac:12:00:04"
            ],
            "name": "9edb93fcacd9",
            "hostname": "9edb93fcacd9",
            "architecture": "x86_64",
            "os": {
              "version": "20.04.4 LTS (Focal Fossa)",
              "family": "debian",
              "name": "Ubuntu",
              "kernel": "4.19.0-18-cloud-amd64",
              "codename": "focal",
              "type": "linux",
              "platform": "ubuntu"
            }
          },
          "cloud": {
            "account": {
              "id": "946960629917"
            },
            "image": {
              "id": "ami-0d90bed76900e679a"
            },
            "instance": {
              "id": "i-03d62e996578c5dc5"
            },
            "machine": {
              "type": "t3.xlarge"
            },
            "region": "us-east-2",
            "availability_zone": "us-east-2c",
            "provider": "aws",
            "service": {
              "name": "EC2"
            }
          },
          "beat": {
            "state": {
              "cluster": {
                "uuid": ""
              },
              "beat": {
                "uuid": "",
                "version": "8.4.0-SNAPSHOT",
                "name": "fleet-server",
                "host": "",
                "type": ""
              }
            }
          },
          "service": {
            "address": "http://unix/state",
            "type": "beat"
          }
        }
      }
    ]
  }
}

kpollich · 2022-07-14T18:05:13Z

Realized I didn't provide a size value for the query above, I checked all 246 documents this time via

GET metrics-elastic_agent.fleet_server-default/_search?size=1000
{
  "_source": ["event.ingested"]
}

No documents in this data stream have an event.ingested field defined, but our unmapped_type setting should be preventing this issue - its seems to work for all other data streams here.

kpollich · 2022-07-14T18:42:39Z

Found this Elasticsearch issue that seems relevant elastic/elasticsearch#81960

kpollich · 2022-07-14T18:58:59Z

The recommendation from the above and its related SDH: https://github.com/elastic/sdh-elasticsearch/issues/5352 was to provide an explicit missing: 0 value for queries that generate this issue. I've added that in 4b2ed43.

I manually made the same change by editing the built .js file inside the E2E Kibana container, restarting Kibana, and re-running the Fleet test suite. After taking those steps, the tests passed. @mdelapenya I kicked off a docker build for this PR.

Since we're green and approved on the Fleet side (thanks @nchaulet). I'm going to merge this PR, and that should unblock the next bump to 8.4 in the E2E repo. Thanks all.

kibana-ci · 2022-07-14T19:53:36Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💚 Build #57770 succeeded 26a89ce
💔 Build #57711 failed 92182ef
💔 Build #57469 failed ce09f35
💚 Build #57105 succeeded 7e07c5d
💚 Build #57036 succeeded a48425d
💔 Build #56807 failed 2187368

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @kpollich

Add ignore_unmapped: true to fix data streams API in e2e tests

ab5206c

kpollich added release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team ci:deploy-cloud v8.4.0 labels Jul 11, 2022

kpollich requested a review from a team as a code owner July 11, 2022 14:47

kpollich self-assigned this Jul 11, 2022

kpollich mentioned this pull request Jul 11, 2022

[automation] update elastic stack version for testing 8.4.0-d0a4da44 elastic/e2e-testing#2771

Merged

nchaulet approved these changes Jul 11, 2022

View reviewed changes

Fix sort setup

dceec04

Merge branch 'main' into fix-data-streams-api-for-e2e-tests

2187368

kpollich added 2 commits July 12, 2022 09:07

Skip failing cypress suite for now

a48425d

Filter out non-integration data streams from Fleet data streams API

7e07c5d

mdelapenya mentioned this pull request Jul 13, 2022

Update elastic-agent-system-metrics deps elastic/beats#32336

Merged

Merge branch 'main' into fix-data-streams-api-for-e2e-tests

dfefbbf

Try using timestamp instead of event.ingested

ce09f35

kpollich changed the title ~~[Fleet] Add ignore_unmapped: true to fix data streams API in e2e tests~~ [Fleet] Use @timestamp instead of event.ingested in data stream sorting Jul 13, 2022

kpollich added 2 commits July 14, 2022 08:29

Merge branch 'main' into fix-data-streams-api-for-e2e-tests

92182ef

Use unmapped type setting

26a89ce

kpollich changed the title ~~[Fleet] Use @timestamp instead of event.ingested in data stream sorting~~ [Fleet] Use unmapped_type: long when sorting datasets that don't include event.ingested Jul 14, 2022

kpollich mentioned this pull request Jul 14, 2022

[Fleet]: Data under Data Streams tab gets removed on running live query under OSQuery tab. elastic/elastic-agent#721

Closed

kpollich requested a review from nchaulet July 14, 2022 17:31

Provide explicit missing value for event.ingested

4b2ed43

nchaulet approved these changes Jul 14, 2022

View reviewed changes

kpollich enabled auto-merge (squash) July 14, 2022 18:59

kpollich changed the title ~~[Fleet] Use unmapped_type: long when sorting datasets that don't include event.ingested~~ [Fleet] Use unmapped_type: long and missing: 0 when sorting datasets that don't include event.ingested Jul 14, 2022

kpollich merged commit 8a5bf42 into elastic:main Jul 14, 2022

kibanamachine added the backport:skip This commit does not require backporting label Jul 14, 2022

kpollich deleted the fix-data-streams-api-for-e2e-tests branch July 14, 2022 20:20

tylersmalley added ci:cloud-deploy Create or update a Cloud deployment and removed ci:deploy-cloud labels Aug 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Use `unmapped_type: long` and `missing: 0` when sorting datasets that don't include `event.ingested` #136114

[Fleet] Use `unmapped_type: long` and `missing: 0` when sorting datasets that don't include `event.ingested` #136114

kpollich commented Jul 11, 2022 •

edited

Loading

elasticmachine commented Jul 11, 2022

kpollich commented Jul 11, 2022

mdelapenya commented Jul 12, 2022 •

edited

Loading

mdelapenya commented Jul 12, 2022 •

edited

Loading

kpollich commented Jul 12, 2022

kpollich commented Jul 12, 2022

mdelapenya commented Jul 13, 2022

mdelapenya commented Jul 13, 2022

kpollich commented Jul 13, 2022

kpollich commented Jul 13, 2022

kpollich commented Jul 13, 2022

kpollich commented Jul 13, 2022

kpollich commented Jul 13, 2022

mdelapenya commented Jul 14, 2022

kpollich commented Jul 14, 2022

kpollich commented Jul 14, 2022

mdelapenya commented Jul 14, 2022

kpollich commented Jul 14, 2022 •

edited

Loading

kpollich commented Jul 14, 2022

kpollich commented Jul 14, 2022

kpollich commented Jul 14, 2022

kpollich commented Jul 14, 2022

kpollich commented Jul 14, 2022 •

edited

Loading

kibana-ci commented Jul 14, 2022

[Fleet] Use unmapped_type: long and missing: 0 when sorting datasets that don't include event.ingested #136114

[Fleet] Use unmapped_type: long and missing: 0 when sorting datasets that don't include event.ingested #136114

Conversation

kpollich commented Jul 11, 2022 • edited Loading

Summary

elasticmachine commented Jul 11, 2022

kpollich commented Jul 11, 2022

mdelapenya commented Jul 12, 2022 • edited Loading

mdelapenya commented Jul 12, 2022 • edited Loading

kpollich commented Jul 12, 2022

kpollich commented Jul 12, 2022

mdelapenya commented Jul 13, 2022

mdelapenya commented Jul 13, 2022

kpollich commented Jul 13, 2022

kpollich commented Jul 13, 2022

kpollich commented Jul 13, 2022

kpollich commented Jul 13, 2022

kpollich commented Jul 13, 2022

mdelapenya commented Jul 14, 2022

kpollich commented Jul 14, 2022

kpollich commented Jul 14, 2022

mdelapenya commented Jul 14, 2022

kpollich commented Jul 14, 2022 • edited Loading

kpollich commented Jul 14, 2022

kpollich commented Jul 14, 2022

kpollich commented Jul 14, 2022

kpollich commented Jul 14, 2022

kpollich commented Jul 14, 2022 • edited Loading

kibana-ci commented Jul 14, 2022

💚 Build Succeeded

Metrics [docs]

History

[Fleet] Use `unmapped_type: long` and `missing: 0` when sorting datasets that don't include `event.ingested` #136114

[Fleet] Use `unmapped_type: long` and `missing: 0` when sorting datasets that don't include `event.ingested` #136114

kpollich commented Jul 11, 2022 •

edited

Loading

mdelapenya commented Jul 12, 2022 •

edited

Loading

mdelapenya commented Jul 12, 2022 •

edited

Loading

kpollich commented Jul 14, 2022 •

edited

Loading

kpollich commented Jul 14, 2022 •

edited

Loading