Elasticsearch 6.4.0 cannot deserialize _ignored from server response of the same version

[erfangc]

Elasticsearch version (bin/elasticsearch --version): 6.4.0

Plugins installed: [kibana, xpack] default Elastic Cloud

JVM version (java -version): Unknown (Elastic Cloud)

OS version (uname -a if on a Unix-like system): Unknown (Elastic Cloud)

Description of the problem including expected versus actual behavior:

Steps to reproduce:

val request = SearchRequest("myindex")
                .source(
                        SearchSourceBuilder()
                                .fetchSource(
                                        arrayOf("field1", "field2"),   emptyArray()
                                )
                                .query(
                                        BoolQueryBuilder()
                                                .minimumShouldMatch(1)
                                                .should(
                                                        MatchPhrasePrefixQueryBuilder("field1", upperCasedStr).boost(1.0f)
                                                )
                                )
                )

This is the stacktrace:

org.elasticsearch.common.xcontent.XContentParseException: [1:224] [innerHitParser] _ignored doesn't support values of type: START_ARRAY
	at org.elasticsearch.common.xcontent.ObjectParser$FieldParser.assertSupports(ObjectParser.java:373)
	at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:167)
	at org.elasticsearch.common.xcontent.ObjectParser.apply(ObjectParser.java:182)
	at org.elasticsearch.search.SearchHit.fromXContent(SearchHit.java:513)
	at org.elasticsearch.search.SearchHits.fromXContent(SearchHits.java:150)
	at org.elasticsearch.action.search.SearchResponse.innerFromXContent(SearchResponse.java:287)
	at org.elasticsearch.action.search.SearchResponse.fromXContent(SearchResponse.java:248)

Here is the actual String response I get:

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 253635,
    "max_score": 1,
    "hits": [
      {
        "_index": "myindex",
        "_type": "_doc",
        "_id": "1",
        "_score": 1,
        "_ignored": [
          "field3",
          "field4"
        ],
        "_source": {
          "field1": "somevalue"
        }
      }
    ]
  }
}

Please include a minimal but complete recreation of the problem, including
(e.g.) index creation, mappings, settings, query etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.

It appears Elasticsearch Java Client 6.4.0 does not understand how to deserialize the _ignored property returned by the server (of the exact same version)

Provide logs (if relevant):

[erfangc]

My Elastic Cloud is running version 6.4.0 and all elasticsearch Java libraries from maven is also 6.4.0 confirmed

[erfangc]

Also, if I am not using the provided Java client and using raw HTTP instead I can simply add a filter_path query parameter to the request. How do I do this in the RestHighLevelClient? Various scouring of the internet reveals nothing useful, though I do see FilterPath.java as a class on one of these Jars

I see that in RequestConverters.java:569 we have:

    private static void addSearchRequestParams(Params params, SearchRequest searchRequest) {
        params.putParam(RestSearchAction.TYPED_KEYS_PARAM, "true");
        params.withRouting(searchRequest.routing());
        params.withPreference(searchRequest.preference());
        params.withIndicesOptions(searchRequest.indicesOptions());
        params.putParam("search_type", searchRequest.searchType().name().toLowerCase(Locale.ROOT));
        if (searchRequest.requestCache() != null) {
            params.putParam("request_cache", Boolean.toString(searchRequest.requestCache()));
        }
        if (searchRequest.allowPartialSearchResults() != null) {
            params.putParam("allow_partial_search_results", Boolean.toString(searchRequest.allowPartialSearchResults()));
        }
        params.putParam("batched_reduce_size", Integer.toString(searchRequest.getBatchedReduceSize()));
        if (searchRequest.scroll() != null) {
            params.putParam("scroll", searchRequest.scroll().keepAlive());
        }
    }

Which severely limits the caller's ability to add their own params such as filter_path for when a bug like the one reported is discovered ... this limits the workaround options for when the SDK is imperfect, I'd recommend that ActionRequest the base class be equip with a flexible way to pass-through URL params (as bugs like the one reported above is highly likely to re-occur.

[elasticmachine]

Pinging @elastic/es-core-infra

[jimczi]

Closing since this bug is already fixed by #32362 which will be available in 6.4.1 when it is released. Thanks for reporting @erfangc .