Search 'fields' option design + implementation

[jtibshirani]

Original issue: #49028
Feature branch: field-retrieval
Docs: https://www.elastic.co/guide/en/elasticsearch/reference/7.x/search-fields.html

Motivation

Often a user wants to retrieve a particular set of fields during a search. Currently, we don't support this usage pattern in a good way. In short, given a list of fields, there is no easy way to load all of their values:

• We can’t load all of them from doc values. Some fields like text fields may not have doc values at all, or we may exceed the limit for a reasonable number of doc value fields to load.
• It’s not easy to load all of them through source. For example, if the field is a field alias, it’s difficult to determine where to find its value in the source.

Better field retrieval support is becoming even more important now that we're introducing more field types that don’t fit the typical pattern like constant_keyword and the proposed runtime fields (#48063).

Feature Summary

We plan to add a new fields section to the search request, which users would specify instead of using source filtering to load fields from source:

POST logs-*/_search
{
  "query": { "match_all": {} },
  "fields": [
    "file.*",
    {
      "field": "event.timestamp",
      "format": "epoch_millis"
    },
    ...
  ]
}

Both full field names and wildcard patterns are accepted. Only leaf fields are returned, the API will not allow for fetching object values. The fields are returned as a flat list in the fields section in each hit, the same as we do for docvalue_fields and script_fields.

Overall, the API gives a friendly way to load fields from source:

• If a non-standard field like a field alias, multi-field, or constant_keyword is specified in fields, then we’ll consult the mappings to find and return the right value.
• The fields are returned in a flat list, as opposed to structured JSON.
• For date and numeric field types, we would support the same format parameter as we do for docvalue_fields to allow for adjusting the format of the results.
• Each value would be returned in a 'canonical' format -- for example if a field is mapped as an integer, it will be returned as an integer even if it was specified as a string in the _source.

Some clarifications:

• In this first pass, the API will not attempt to load from stored fields or doc values.
• For simplicity of parsing, values will always be returned in an array, even if there is only one value present.

Implementation Plan

• Introduce a fields section in the search request that fetches values from source. (Add a simple 'fetch fields' phase. #55639)
• Correctly resolve field aliases, multi-fields, and copy_to. (In field retrieval API, handle non-standard source paths. #55889)
• Consult field mappings to parse and correctly format each value. Also handle constant_keyword. (Allow field mappers to retrieve fields from source. #56928)
• Handle ignore_malformed. (Allow field mappers to retrieve fields from source. #56928)
• Support ignore_above. (Fix casting of scaled_float in sorts (backport of #57207) #57385)
• Support setting a format through the format parameter. (Deprecte Rounding#round (backport #57845) #57893)
• Measure performance and look into improvements. (For the fields fetch phase, avoid reloading stored fields. #58196)
• Support null_value. (Respect null_value parameter in the fields retrieval API. #58623)
• Improve documentation around field loading. (Add a reference on returning fields during a search. #57500, Add docs for the fields retrieval API. #58787)
• Handle geo values. (Support spatial fields in field retrieval API. #59821)

Future improvements:

• Move FieldMapper#lookupValues to MappedFieldType. (?)
• Handle meta fields like _size.
• Make use of more efficient source parsing: Partially parse source documents to speed up source access #52591.
• Support the API in inner_hits.

Open Questions

• If a wildcard pattern matches both a parent field and one of its multi-fields, should we just return the parent to avoid returning the same value twice? A similar question holds for field aliases and their target fields.
• Should the API return fields in _source that have been disabled in the mappings (enabled: false)?
• For keyword fields, should we apply the normalizer or return the original value?

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[jtibshirani]

I thought more about the question of whether we should apply normalizer before returning a keyword, and to me it makes sense to apply normalization by default:

• The actual name normalizer suggests that it is performing standardization, which is exactly the sort of thing we claim to do when returning values. I was hesitant earlier because normalization felt more like text analysis to me, but I think it's typically value standardization (and we just happen to re-use the analyzer framework).
• It's most consistent with other parts of the search response, in particular terms aggregations.
• If the original value is actually interesting in terms of casing, accents, etc., then a user will probably avoid normalizing, and instead perform a case-insensitive search (coming soon :)). Or, they would set up a multi-field where one field contains the original, and the other a normalized form.

Tagging @jpountz @jimczi @javanna @nik9000 in case they have thoughts on the above, happy to discuss here!

[jimczi]

+1 to support normalizer ootb. As you said it's consistent with aggregations and script. That would also make the docvalue_fields alternative less appealing, which I find important since we want to simplify the reasoning when retrieving fields.

[javanna]

@jtibshirani when you say "by default" does that mean it can be disabled? From the point of view of "when you retrieve, you get what you sent, when you search and aggregate, you search on and get back what was indexed" I am torn, I would personally expect the raw value loaded from source. Though if users can control what they get, I would not mind that the default is normalized.

[jtibshirani]

@javanna I was indeed wondering if we could make the behavior configurable, but don't have immediate plans to do so. It's always nice to avoid configuration options and have strong defaults.

I have also been on the fence about this, I can see arguments both ways. I suggest that we move forward with normalizing the values for now. I'm going to ask the teams planning to use this feature (SQL, ML, Kibana) to try to integrate with it before we ship it, and have a short list of questions I plan to ask them which includes keyword normalization. The questions are tracked in the issue description.

[jtibshirani]

We discussed how geo fields should be returned with @talevy and @imotov. A summary of our discussion:

• Geo data should be returned in a consistent format. We accept a variety of formats during indexing, and feel it would be helpful for clients if all fields were returned in a single format.
• The default format should be 'geojson', since this matches our usual JSON return format, and it's natural for Kibana. We should also support well-known text, which is best for SQL. Note that geo points (in addition to shapes) will also be returned in geojson.
• The user will be able to select the format by setting format: wkt or format: geojson.
• Ideally if the source value is already in the right format to return, we won't re-parse it to a geolib object then re-serialize it.

[jtibshirani]

An additional note: now that we'll return points in geojson format by default, for consistency we should accept this format when indexing points. We currently don't allow geojson, the work to add support is tracked in #47815.

[jtibshirani]

The feature branch was merged in #60100. I'll open new issues/ PRs for the follow-up improvements mentioned in the description.