Breaking change in 7.x.y for date format

[faxm0dem]

During my upgrade from 6.8.6 to 7.5.1 I discovered that the date formats don't parse anymore.
There's a difference in how timezones are parsed (Z).

In 6.8.6 this works:

6.8.6☠ PUT test
HTTP/1.1 200 OK
content-length: 63
content-type: application/json; charset=UTF-8

{
    "acknowledged": true,
    "index": "test",
    "shards_acknowledged": true
}

6.8.6☠ PUT test/_doc/_mapping <<<'{"properties":{"time":{"type":"date","format":"'"YYYY-MM-dd'T'HH:mm:ssZ"'"}}}'
HTTP/1.1 200 OK
Warning: 299 Elasticsearch-6.8.6-3d9f765 "[types removal] Specifying types in put mapping requests is deprecated. To be compatible with 7.0, the mapping definition should not be nested under the type name, and the parameter include_type_name must be provided and set to false."
content-length: 21
content-type: application/json; charset=UTF-8

{
    "acknowledged": true
}

6.8.6☠ POST test/_doc <<<'{"time":"2019-01-23T12:10:00+01:00"}'
HTTP/1.1 201 Created
Location: /test/_doc/697C1m8Bzi_qg7-P99RF
content-length: 171
content-type: application/json; charset=UTF-8

{
    "_id": "697C1m8Bzi_qg7-P99RF",
    "_index": "test",
    "_primary_term": 1,
    "_seq_no": 0,
    "_shards": {
        "failed": 0,
        "successful": 2,
        "total": 2
    },
    "_type": "_doc",
    "_version": 1,
    "result": "created"
}

In 7.5.1 it fails:

7.5.1☠ PUT test
HTTP/1.1 200 OK
content-length: 63
content-type: application/json; charset=UTF-8

{
    "acknowledged": true,
    "index": "test",
    "shards_acknowledged": true
}

7.5.1☠ PUT test/_mapping <<<'{"properties":{"time":{"type":"date","format":"'"YYYY-MM-dd'T'HH:mm:ssZ"'"}}}'
HTTP/1.1 200 OK
content-length: 21
content-type: application/json; charset=UTF-8

{
    "acknowledged": true
}

7.5.1☠ POST test/_doc <<<'{"time":"2019-01-23T12:10:00+01:00"}'
HTTP/1.1 400 Bad Request
content-length: 720
content-type: application/json; charset=UTF-8

{
    "error": {
        "caused_by": {
            "caused_by": {
                "reason": "date_time_parse_exception: Text '2019-01-23T12:10:00+01:00' could not be parsed at index 19",
                "type": "date_time_parse_exception"
            },
            "reason": "failed to parse date field [2019-01-23T12:10:00+01:00] with format [YYYY-MM-dd'T'HH:mm:ssZ]",
            "type": "illegal_argument_exception"
        },
        "reason": "failed to parse field [time] of type [date] in document with id '_enA1m8BLwBLCwEtLLjx'. Preview of field's value: '2019-01-23T12:10:00+01:00'",
        "root_cause": [
            {
                "reason": "failed to parse field [time] of type [date] in document with id '_enA1m8BLwBLCwEtLLjx'. Preview of field's value: '2019-01-23T12:10:00+01:00'",
                "type": "mapper_parsing_exception"
            }
        ],
        "type": "mapper_parsing_exception"
    },
    "status": 400
}

After some trial and error, I discovered that I needed to specify as many Z as tere were characters for the timezone. So this format works in 7.5.1: YYYY-MM-dd'T'HH:mm:ssZZZZZ, but it doesn't in 6.8.6.

[faxm0dem]

I have to add that I used java 1.8 in both cases, not the bundled java11

[elasticmachine]

Pinging @elastic/es-search (:Search/Mapping)

[faxm0dem]

In fact, using YYYY-MM-dd'T'HH:mm:ssZZZZZ in the date format doesn't throw, but it indexes all my current documents to the correct time, but to January first 1970...

[faxm0dem]

Update: I realised I was using the weekyear Y instead of the "era" year y.
If I use the yyyy-MM-dd'T'HH:mm:ssZZZZZ, the document is correctly parsed.
So I guess it accidentally worked in 6.x

[mayya-sharipova]

@faxm0dem Just to confirm, does the code on 7.5 works as expected and this issue can be closed?

[pgomulka]

@faxm0dem thank you for raising this.
This is indeed a breaking change and it is documented in our breaking changes list for 7.0 https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html#breaking_70_java_time_changes

As described on that page, we are using java.time implementation for date parsing and formatting (previously using joda)
You were affected by two of these.
Y - year-of-era in 6.8 becomes y in java.time and ES 7
Z has similar meaining but differs when it comes to number of letters used and parsing of 0offset (Zulu)
as per https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html

Offset Z: This formats the offset based on the number of pattern letters. One, two or three letters outputs the hour and minute, without a colon, such as '+0130'. The output will be '+0000' when the offset is zero. Four letters outputs the full form of localized offset, equivalent to four letters of Offset-O. The output will be the corresponding localized offset text if the offset is zero. Five letters outputs the hour, minute, with optional second if non-zero, with colon. It outputs 'Z' if the offset is zero. Six or more letters throws IllegalArgumentException.

I don't know your usecase but you probably want to use XXX

Offset X and x: This formats the offset based on the number of pattern letters. One letter outputs just the hour, such as '+01', unless the minute is non-zero in which case the minute is also output, such as '+0130'. Two letters outputs the hour and minute, without a colon, such as '+0130'. Three letters outputs the hour and minute, with a colon, such as '+01:30'. Four letters outputs the hour and minute and optional second, without a colon, such as '+013015'. Five letters outputs the hour and minute and optional second, with a colon, such as '+01:30:15'. Six or more letters throws IllegalArgumentException. Pattern letter 'X' (upper case) will output 'Z' when the offset to be output would be zero, whereas pattern letter 'x' (lower case) will output '+00', '+0000', or '+00:00'.

You might be also interested in https://www.elastic.co/guide/en/kibana/6.8/upgrade-assistant.html
This tool can identify all the breaking changes in mappings you are using. You can then create new with fixed pattern them and perform reindex.

You can find more backgorund on this here https://www.elastic.co/blog/journey-support-nanosecond-timestamps-elasticsearch

[faxm0dem]

Thanks for your input, I'm closing this then