Values above ignore_above mapping limit don't show up in Discover anymore

[andrewkroh]

Kibana version: 7.13.1

Elasticsearch version: 7.13.1

Server OS version: Elastic Cloud hosted

Browser version: Firefox 89.0

Browser OS version: macOS

Original install method (e.g. download page, yum, from source, etc.):

Describe the bug:

The event.original field is missing in Kibana Discover for some events.

Steps to reproduce:

1. Ingest data using a Beat that creates event.original.
2. Open discover and look at the field event.original. Note that some events appear to be missing event.original.
3. Get one of the event's _id and query it directly from Elasticsearch and it will have event.original.
4. I'm attaching a file that contains both the Kibana discover JSON and the direct ES query output.

missing-event.original.txt

Expected behavior:

event.original would be displayed. Or if it is being dropped for some reason, such as length, that there would be an indication that this is happening.

One interesting data point is that querying for _id:aKSBznkBdD5BTT25BM6w and event.original:* returns no results so even the KQL query seems to think that event.original does not exist despite aKSBznkBdD5BTT25BM6w having an event.original.

Screenshots (if relevant):

Errors in browser console (if relevant): There are none.

Provide logs and/or server output (if relevant):

Any additional context:

I'm not sure if it has something to do with the length of the field or possibly the field's content. I saw this in cases where the event.original contained both JSON and in a Windows event log use case where event.original contained a big XML blob.

My workaround has been to modify Kibana advanced settings to turn on discover:searchFieldsFromSource (this will affect runtime fields and fields not in _source).

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[andrewkroh]

The mapping for event.original has ignore_above: 1024. The theory is that because it's not indexed that it doesn't show up in the fields that are returned by the query. The docs do say that it honors ignore_above.

        "event": {
          "properties": {

            "original": {
              "type": "keyword",
              "index": false,
              "ignore_above": 1024
            },

I can replicate what I was seeing initially in Discover with this query. fields does not contain the event.original for the long values.

GET journalbeat-8.0.0-2021.05.07-000013/_search
{
  "query": {
    "term": {"_id": "aKSBznkBdD5BTT25BM6w"}
  },
  "fields": [
    "*"
  ]
}

I think this will be mitigated by #95946 (comment). And we could bump the ignore_above slightly for some Fleet integrations that are expected to have larger event.original values.

[kertal]

@andrewkroh thx for opening this issue, we will investigate

[timroes]

Blocked on: elastic/elasticsearch#74121

[timroes]

Discussing the desired behavior with PMs, we came to the following conclusion: We want to show ignored values (e.g. because they are above ignore_above) in the Discover table in columns and expanded documents directly. We'd additionally want to add a warning icon with a tooltip beside them explaining that those values are not indexed, and thus cannot be searched through.