Bug - Date field used as sort option can crash ES with BigInteger error #48074

DemonGyro · 2019-10-15T16:39:38Z

Elasticsearch version (bin/elasticsearch --version): 7.3.1

Plugins installed: [discovery-ec2, repository-s3]

JVM version (java -version):
java version "11.0.4" 2019-07-16 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.4+10-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.4+10-LTS, mixed mode)

OS version (uname -a if on a Unix-like system):
Linux XXXXXXXXX 4.15.0-1051-aws #53-Ubuntu SMP Wed Sep 18 13:35:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
The date value returned in the sort field returned in a document from _search had a value that was invalid. When this was passed back in to the search_after field, this caused the request to fail with a java.lang.AssertionError: Unknown number type []BIG_INTEGER error. With enough errors, this ended up crashing ES. We were seeing this take down an entire cluster with only a couple requests.

The values returned from the sort field of a document from _search are expected to be valid and are expected to not cause crashes in the ES application.

Steps to reproduce:

Create an index that has a date.

PUT _template/test_index
{
  "index_patterns": [
    "test*"
  ],
  "mappings": {
    "test": {
      "properties": {
        "test": {
          "type": "boolean"
        },
        "created": {
          "type": "date"
        }
      }
    }
  }
}

Add about a dozen documents. 8 good, 4 bad.

Good example:

POST /test_0/test
{
  "test": true,
  "created": "2017-12-20T15:42:55.6497236+00:00"
}

Bad example:

POST /test_0/test
{
  "test": true
}

Execute a search on the date field as sort

POST /_search
{
    "query": {
    	"match_all": {}
    },
    "sort": [
        {
            "created": {
                "order": "asc"
            }
        }
    ]
}

If done correctly, the last item in the search should look like this:

{
        "_index": "test_b10",
        "_type": "test",
        "_id": "OSA_0G0BUWBv2lD0FAe4",
        "_score": null,
        "_source": {
          "test": true
        },
        "sort": [
          9223372036854776000
        ]
}

Execute a search with the last item's sort value as the search_after field.

{
    "query": {
        "match_all": {}
    },
    "sort": [
        {
            "created": {
                "order": "asc"
            }
        }
    ],
    "search_after": [
        9223372036854776000
    ]
}

Result from search should be a 500 error.

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_state_exception",
        "reason": "No matching token for number_type [BIG_INTEGER]"
      }
    ],
    "type": "illegal_state_exception",
    "reason": "No matching token for number_type [BIG_INTEGER]"
  },
  "status": 500
}

Stack Trace

[2019-10-10T20:34:56,297][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [ES_NODE_NAME] fatal error in thread [Thread-6470], exiting
java.lang.AssertionError: Unknown number type []BIG_INTEGER
        at org.elasticsearch.search.searchafter.SearchAfterBuilder.fromXContent(SearchAfterBuilder.java:247) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.search.builder.SearchSourceBuilder.parseXContent(SearchSourceBuilder.java:1189) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.rest.action.search.RestSearchAction.parseSearchRequest(RestSearchAction.java:131) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.rest.action.search.RestSearchAction.lambda$prepareRequest$1(RestSearchAction.java:110) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.rest.RestRequest.withContentOrSourceParamParserOrNull(RestRequest.java:449) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.rest.action.search.RestSearchAction.prepareRequest(RestSearchAction.java:109) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:92) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:69) ~[?:?]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:344) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:320) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:370) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:299) ~[elasticsearch-7.3.1.jar:7.3.1]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:66) ~[?:?]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:31) ~[?:?]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[?:?]
        at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) ~[?:?]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:835) [?:?]

The text was updated successfully, but these errors were encountered:

elasticmachine · 2019-10-15T19:14:04Z

Pinging @elastic/es-search (:Search/Search)

jimczi · 2019-10-15T20:05:52Z

It seems that your library rewrites the json response since the raw response should contain 9223372036854775807 for documents that don't have a value for the sorted field. This value is the maximum number that a long can encode (2^63 - 1). However some library transforms any numeric in json as a floating point number and in this case the long value 9223372036854775807L is represented as the double 9223372036854776000. Since this value is greater than the maximum value that a long can encode, the Elasticsearch parser tries to load it with a BigInteger but search_after complains that he's expecting a long.
I hope that you don't mind if I close this issue but we try to keep github for verified bugs and feature requests and this problem is not caused by Elasticsearch itself but rather by the library that you use to communicate with the server. Please open a new discussion in the forum if you have any follow up questions.

fperks · 2019-10-16T15:09:06Z

@jimczi

Can you please explain? Doing the above causes Elasticsearch service to terminate. Sending bad data should not terminate the ES service with a fatal exception. This is not an issue with a client as i can reproduce it using curl. Furthermore this was not an issue on 6.x, and only seems to be an issue after we updated to 7.x.

Doing this:

curl -X POST \
  http://x.x.x.x:9200/_search \
  -H 'Content-Type: application/json' \
  -H 'cache-control: no-cache' \
  -d '{
    "query": {
        "match_all": {}
    },
    "sort": [
        {
            "created": {
                "order": "asc"
            }
        }
    ],
    "search_after": [
        922337203685477600012718
    ]
}'

Should not lead to this:

Oct 16 15:04:40 ip-x-x-x-x systemd[1]: Started Elasticsearch.
Oct 16 15:04:46 ip-x-x-x-x elasticsearch[3738]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Oct 16 15:05:30 ip-x-x-x-x systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Oct 16 15:05:30 ip-x-x-x-x systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

[2019-10-16T14:51:43,569][ERROR][o.e.ExceptionsHelper     ] [IP_X_data_i3large] fatal error
	at org.elasticsearch.ExceptionsHelper.lambda$maybeDieOnAnotherThread$4(ExceptionsHelper.java:300)
	at java.base/java.util.Optional.ifPresent(Optional.java:183)
	at org.elasticsearch.ExceptionsHelper.maybeDieOnAnotherThread(ExceptionsHelper.java:290)
	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.exceptionCaught(Netty4HttpRequestHandler.java:76)
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:297)
	at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:831)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:376)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
	at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.base/java.lang.Thread.run(Thread.java:835)
[2019-10-16T14:51:43,574][WARN ][o.e.h.AbstractHttpServerTransport] [IP_X_data_i3large] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/10.0.81.117:9200, remoteAddress=/x.x.x.x:53246}
java.lang.Exception: java.lang.AssertionError: Unknown number type []BIG_INTEGER
	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.exceptionCaught(Netty4HttpRequestHandler.java:79) [transport-netty4-client-7.3.0.jar:7.3.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:297) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:831) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:376) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) [transport-netty4-client-7.3.0.jar:7.3.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [netty-codec-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) [netty-handler-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.36.Final.jar:4.1.36.Final]
	at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: java.lang.AssertionError: Unknown number type []BIG_INTEGER
	at org.elasticsearch.search.searchafter.SearchAfterBuilder.fromXContent(SearchAfterBuilder.java:247) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.search.builder.SearchSourceBuilder.parseXContent(SearchSourceBuilder.java:1189) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.rest.action.search.RestSearchAction.parseSearchRequest(RestSearchAction.java:131) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.rest.action.search.RestSearchAction.lambda$prepareRequest$1(RestSearchAction.java:110) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.rest.RestRequest.withContentOrSourceParamParserOrNull(RestRequest.java:449) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.rest.action.search.RestSearchAction.prepareRequest(RestSearchAction.java:109) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:92) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:344) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:320) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:370) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:299) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:66) ~[?:?]
	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:31) ~[?:?]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
	... 40 more
[2019-10-16T14:51:43,587][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [IP_X_data_i3large] fatal error in thread [Thread-11661], exiting
java.lang.AssertionError: Unknown number type []BIG_INTEGER
	at org.elasticsearch.search.searchafter.SearchAfterBuilder.fromXContent(SearchAfterBuilder.java:247) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.search.builder.SearchSourceBuilder.parseXContent(SearchSourceBuilder.java:1189) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.rest.action.search.RestSearchAction.parseSearchRequest(RestSearchAction.java:131) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.rest.action.search.RestSearchAction.lambda$prepareRequest$1(RestSearchAction.java:110) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.rest.RestRequest.withContentOrSourceParamParserOrNull(RestRequest.java:449) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.rest.action.search.RestSearchAction.prepareRequest(RestSearchAction.java:109) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:92) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:344) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:320) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:370) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:299) ~[elasticsearch-7.3.0.jar:7.3.0]
	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:66) ~[?:?]
	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:31) ~[?:?]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
	at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[?:?]
	at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:835) [?:?]

jimczi · 2019-10-16T15:19:08Z

I read the issue too quickly, sorry. We accept BigInteger now in xcontent but this code was not updated. This is indeed a bug, hence reopening.

The support for BigInteger and BigDecimal was added for XContent in elastic#32888. However the SearchAfterBuilder xcontent parser doesn't expect them to be present so it throws an AssertionError. This change fixes this discrepancy by changing the AssertionError into an IllegalArgumentException that will not cause the node to die when thrown. Closes elastic#48074

jimczi · 2019-10-16T15:59:34Z

@fperks thanks, I opened #48147 to fix this bug.

fperks · 2019-10-16T16:01:28Z

Thank you @jimczi !

* Do not throw errors on unknown types in SearchAfterBuilder The support for BigInteger and BigDecimal was added for XContent in #32888. However the SearchAfterBuilder xcontent parser doesn't expect them to be present so it throws an AssertionError. This change fixes this discrepancy by changing the AssertionError into an IllegalArgumentException that will not cause the node to die when thrown. Closes #48074

$@polyfractal$ polyfractal added the :Search/Search Search-related issues that do not fall into other categories label Oct 15, 2019

$@polyfractal$ polyfractal added the >bug label Oct 15, 2019

jimczi closed this as completed Oct 15, 2019

jimczi reopened this Oct 16, 2019

jimczi mentioned this issue Oct 16, 2019

Do not throw errors on unknown types in SearchAfterBuilder #48147

Merged

jimczi closed this as completed in #48147 Oct 23, 2019

This was referenced Feb 3, 2020

[meta] 7.6 release elastic/elasticsearch-net#4340

Closed

[meta] 7.6 release elastic/elasticsearch-net#4341

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug - Date field used as sort option can crash ES with BigInteger error #48074

Bug - Date field used as sort option can crash ES with BigInteger error #48074

DemonGyro commented Oct 15, 2019 •

edited

Loading

elasticmachine commented Oct 15, 2019

jimczi commented Oct 15, 2019

fperks commented Oct 16, 2019

jimczi commented Oct 16, 2019

jimczi commented Oct 16, 2019

fperks commented Oct 16, 2019

Bug - Date field used as sort option can crash ES with BigInteger error #48074

Bug - Date field used as sort option can crash ES with BigInteger error #48074

Comments

DemonGyro commented Oct 15, 2019 • edited Loading

elasticmachine commented Oct 15, 2019

jimczi commented Oct 15, 2019

fperks commented Oct 16, 2019

jimczi commented Oct 16, 2019

jimczi commented Oct 16, 2019

fperks commented Oct 16, 2019

DemonGyro commented Oct 15, 2019 •

edited

Loading