Consider adding field alias information to field caps.

[jtibshirani]

When used in a search, field aliases have some limitations compared to concrete fields (as described in unsupported APIs). One important limitation is that field aliases do not appear in the document _source, and so their values cannot be retrieved through _source filtering.

For this reason, applications may need to add special handling for field aliases when working with the _source from search results. Currently, the _field_caps API presents a completely transparent view of field aliases -- requesting field caps for an alias will return the same output as if it were mapped as a concrete field. One proposal is to include a new aliases section that indicates the field is an alias, and gives information about its target:

{
  "indices" : [ "index1", "index2" ],
  "fields" : {
    "alias_field" : {
      "keyword" : {
        "type" : "keyword",
        "searchable" : true,
        "aggregatable" : true
      }
    }
  },
  "aliases": {
    "alias_field": [{
      "path": "concrete_field",
      "indices": [ "index2" ]
    }]
  }
}

To me this seems like an acceptable addition to field caps and is not just creating a 'leaky abstraction'. The fact that a field is an alias has an impact on its capabilities, and an application may want to add special handling for this case.

Motivation

ES SQL currently loads field values from doc_values, but is exploring loading some values from _source (#44062). If a field alias is specified, SQL needs to understand that it is an alias in order to either fall back to doc_values, or resolve it to its target in order to extract its value from _source. ES SQL uses field caps to retrieve information about fields across indices/ clusters, and does not have knowledge of the original field mappings.

In Kibana's Discover app, field aliases currently do not show up on the left-hand panel or in the body of search results, because this data is pulled directly from _source. Having knowledge of field aliases in the index pattern could allow Kibana to show the aliases associated with concrete fields in the documents (elastic/kibana#28570). Kibana index patterns pull information from field caps, so this could be a helpful addition to the API.

Alternative approaches

Another approach would be to try to take field aliases into account when filtering and returning _source. To me this seems more confusing from an API perspective:

• Would filtering on "my_object.*" look up matching field aliases and try to include their targets in the returned _source? Or is it interpreted as just taking the _source and filtering out the parts that don't match "my_object.*"?
• It's not clear how field alias information could be returned as part of the _source. We do not want to modify or rewrite _source because it should represent exactly what was provided when indexing the document.

[elasticmachine]

Pinging @elastic/es-search

[jtibshirani]

Tagging @elastic/kibana-app-arch and @elastic/es-sql for their thoughts on the proposal.

[jpountz]

We have the same issues with multi fields and copy_to targets. copy_to is hard to address since multiple source fields can have the same target, but maybe we could have a solution that also addresses the issue with multi fields by returning the path of the field in the _source document, e.g.

{
  "indices" : [ "index1", "index2" ],
  "fields" : {
    "alias_field" : {
      "keyword" : {
        "type" : "keyword",
        "searchable" : true,
        "aggregatable" : true,
        "source_path": "concrete_field"
      }
    },
    "some_field.multi_field" : {
      "keyword" : {
        "type" : "keyword",
        "searchable" : true,
        "aggregatable" : true,
        "source_path": "some_field"
      }
    }
  }
}

I think the same information could be used internally to support highlighting on aliases and multi-fields?

[ruflin]

I can't speak for the Kibana team but I think both approaches could work. The part I like about the approach proposed by @jpountz is that it seems to be just an extension of the existing Metadata about the field and it works not only for alias.

[jimczi]

We discussed this issue in the Search meeting and decided to implement the approach proposed by Julie and Adrien (adding source_path to the output of the field_caps API). I opened #49264 to track the progress so I am closing this one now.