[Fleet] No matching data streams found on Fleet Management Page

[nicpenning]

Kibana version:
8.3.2
Elasticsearch version:
8.3.2
Server OS version:
Windows Server 2019
Browser version:
Chrome Version 103.0.5060.114 (Official Build) (64-bit)
Browser OS version:
Windows 11
Original install method (e.g. download page, yum, from source, etc.):
Windows / On-Premise / ZIP
Describe the bug:
When visiting the Fleet management page and visiting the Data Streams page, nothing shows up:

Steps to reproduce:

1. Be on 8.3.2 of the Elastic Stack
2. Navigate to Fleet -> Data Streams
3. See no Data Streams show up
4. Check Dev Tools in browser to see error.

Expected behavior:
I expect this to show all of the data streams :)

Screenshots (if relevant):
Screenshots above

Errors in browser console (if relevant):
The request:

The response is:

{"statusCode":500,
"error":"Internal Server Error",
"message":"search_phase_execution_exception: [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [data_stream.dataset] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}

[nicpenning]

By checking all of our data stream mappings, we found a few indices .ds* that were custom datastreams that were using the data_stream.dataset type as keyword which was different than most of the other datastreams which were set to type constant_keyword, so intuitively we went ahead and checked the templates and it appeared we did not have the data_stream ECS mappings set. Fortunately there was a component template that was for datastream ECS fields that conveniently had the appropriate field mappings we needed so we add that to the data streams that were not compliant. We then proceeded to rollover the datastreams the. Reindeer the old data unto the new datastreams. There was over 50GB in each datastream. After I removed 1 of the offending datastreams after it was reindexed, then the datastreams were able to populate in the Fleet -> Data Streams section as noted above.

Strange that the other datastream, which hadn't been deleted since it was still rendering, still existed with no problems. Perhaps the rollover with the proper mapping was enough for the datastreams api call to work? We are unsure.

If anything comes of this, it would be great to populate an error in the UI saying that there are some incorrect mappings for the specific datastreams and how to resolve it. 😀

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[hop-dev]

I've tried recreating this unsuccessfully so far, I think this is representative:

PUT _component_template/bad-mapping5
{
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date",
          "format": "date_optional_time||epoch_millis"
        },
        "message": {
          "type": "text"
        },
        "data_stream": {
          "properties": {
            "dataset": { "type" : "keyword"},
            "namespace": { "type" : "constant_keyword"},
            "type": { "type" : "constant_keyword"}
          }
        }
      }
    }
  },
  "_meta": {
    "description": "bad mapping for data_stream fields",
    "managed": true,
    "managed_by": "fleet"
  }
}

PUT _index_template/my-index-template5
{
  "index_patterns": ["metrics-bad_mapping5-*"],
  "data_stream": {},
  "composed_of": [ "bad-mapping5" ],
  "priority": 500,
  "_meta": {
    "description": "bad mapping for data_stream fields",
    "managed": true,
    "managed_by": "fleet"
  }
}

POST /metrics-bad_mapping5-default/_doc
{
  "@timestamp": "2022-01-01",
  "message": "hello world",
  "data_stream": {
    "dataset": "bad_mapping5",
    "namespace": "default",
    "type": "metrics"
  }
}

But this is successfully shown with no error:

[nicpenning]

We had undergone many upgrades since early 7.0 days so perhaps this is an edge case for those who have gone through many upgrades and built custom data streams that could conflict. I really don't out side the facts above. Sorry, I can't be of more help.

[hop-dev]

Thinks nic. I tried a few ways of recreating this, variations of the above commands with different field types, or mixing field types on backing indices for a dat stream and the API remained functioning. I guess that the upgrades is causing something quite niche but I can't see any low hanging fruit to fix here, although I agree this API is slightly brittle by nature.

@kpollich I think we may want to close this unless we see it becoming more common

[nicpenning]

Please see this conversation on the slack community:
https://elasticstack.slack.com/archives/CND4DUJ10/p1658152613492109?thread_ts=1658152613.492109&cid=CND4DUJ10

It shows another user with the same problem.

[criamico]

We had undergone many upgrades since early 7.0 days so perhaps this is an edge case for those who have gone through many upgrades and built custom data streams that could conflict.
Yes I think that this might be the case, I recently worked on another bug on the data streams page that was caused by old indices that remain after a major upgrade. For some reason the rollover doesn't happen and can cause unexpected bugs.

The error message:

{"statusCode":500,
"error":"Internal Server Error",
"message":"search_phase_execution_exception: [illegal_argument_exception] Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [data_stream.dataset] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}

is the same as #138908, which was filed roughly in the same period. I see that this other bug is also open, but I wonder how often are they occurring now. The data stream page has gone through many changes in the past six months so I'm not even sure that these bugs are reproducible now. I tried to reproduce this one with no success. @jlind23 can we close this and reopen if we get new occurrences?

--

I also checked the message in the slack community, that was posted roughly 6 months ago:

{"statusCode":500,"error":"Internal Server Error","message":"search_phase_execution_exception: [illegal_argument_exception] Reason: Field [event.ingested] of type [keyword] is not supported for aggregation [max]"}

I think that this might be an occurrence of elastic/elastic-agent#654 or similar issue. The slack message doesn't say which version of the stack they were running, so it's difficult to know what's the exact case.

[jlind23]

@criamico Thanks for looking into this. I will close it as won't fix and reopen afterwards if needed.