[Logs UI] The UI throws error when @timestamp format is date_nanos

[simianhacker]

Kibana version:
master

Elasticsearch version:
master

Describe the bug:
When @timestamp is set to date_nanos the date formatter throws an error.

Steps to reproduce:

1. Index some logs setting @timestamp to date_nanos (contact @simianhacker for help)
2. Open Logs UI
3. :(

Expected behavior:
It should render properly without errors.

Screenshots (if relevant):

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[weltenwort]

related: #40183

[jasonrhodes]

Note: from @weltenwort -- Discover and some other places in Kibana have solved how to handle the nanosecond format for bigint, we should look into those implementations for guidance on how to fix this in Logs UI.

[joshdover]

@Kerry350 Has there been any progress on this? We are exploring supporting nanoseconds in Beats/Agent and would need to be sure Kibana fully supports this.

[Kerry350]

Hi @joshdover, there hasn't been any progress on this to date. Do you know what the timeframe would be for adding the support in Beats / Agent? This would allow us to try and reprioritise the support.

[jughosta]

If it's of any help here, there is a way of getting date nanos as a string instead of a timestamp so it can be safely used for search_after. Example #157241

[ruflin]

Adding some more links for context here:

• Discover nano support: [OnWeek][Discover] Allow to fetch more documents on Discover page #163784
• Beats adding support for nano seconds: Add support for nanosecond timestamps beats#15871 / Add support for nanosecond precision timestamps beats#31553

To reproduce it, here some Dev Tools command:

DELETE _index_template/foo

PUT _index_template/foo
{
  "index_patterns": ["logs-foo-bar"],
  "template": {
    "mappings": {
      "properties": {
        "host_name": {
          "type": "keyword"
        },
        "@timestamp": {
          "type": "date_nanos"
        }
      }
    }
  },
  "priority": 500,
  "version": 3,
  "data_stream": { }
}

DELETE _data_stream/logs-foo-bar
POST logs-foo-bar/_doc
{
  "@timestamp": "2023-09-07T15:04:05.000000002Z",
  "message": "Second message"
}

POST logs-foo-bar/_doc
{
  "@timestamp": "2023-09-07T15:04:05.000000001Z",
  "message": "First message"
}

POST logs-foo-bar/_doc
{
  "@timestamp": "2023-09-07T15:04:05.000000003Z",
  "message": "Third message"
}

POST logs-foo-bar/_doc
{
  "@timestamp": "2023-09-07T15:04:05.000000000Z",
  "message": "Third message"
}

GET logs-foo-bar/_search
{
  "sort" : ["@timestamp"]
}

Going to Discover and include the time range, the sorting works as expected. Going to Logs UI and including the time range with the timestamp above shows the error

[weltenwort]

Some thoughts and hints for anyone picking this up in the future:

As @jughosta pointed out we need to fetch the timestamp as a string because a numeric nanosecond timestamp exceeds the value range of normal ECMAScript numbers. In contrast to Discover, the Logs UI interprets the timestamp in several places, though, and even performs date math on it.

So while switching the format parameter in queries to strict_date_optional_time_nanos and the TypeScript type from number to string should work in many places where we just pass the value back to Elasticsearch, some usage sites might have to be rewritten to keep working. A quick list off the top of my head would be:

• The "fetch around" logic, in which we add 1 ms as the smallest step to not skip any documents with search_after.
• How we pass the timestamp to the minimap to mark the visible window.
• How we pass the timestamp to ML during job creation.
• How we sync the position to the URL.

[mohamedhamed-ahmed]

I timeboxed this ticket and quickly tried an approach similar to the one discover applied using strict_date_optional_time_nanos. The solution works and the UI was working fine as in the video below.

Adding to what @weltenwort mentioned and the places that need to be modified we also need to take care of:

1. The logs flyout as it is also using the @timestamp value.
2. The test files
3. Modifying the types as they are propagating in many places in the code.

I wasn't able to spend much time to be more precise on exact expected time that we need to get this fixed but the change doesn't seem to be a quick one.

One question though, do we just need the UI to not break with date_nanos, or do we also need to display and make use of the date_nanos in the Stream page?

Because one option we might have is that we retrieve the timestamp as date_nanos but remove the precision of the nano_seconds when we get the response and continue using it in number as before. Sounds a bit hacky and not sure about the consequences of this in the code until we spend more time into it but could be an option here.

date_nanos.mov

[ruflin]

One question though, do we just need the UI to not break with date_nanos, or do we also need to display and make use of the date_nanos in the Stream page?

I think step one should be to get the Logs UI not working when there is a nano timestamp in the data. If there is a quick fix to this through rounding, I suggest we do it and follow up with the more detailed fix in step 2.

[mohamedhamed-ahmed]

After discussing this further with @weltenwort we concluded that it would be best to solve this in a proper way and address all places where the timestamp is being used. Otherwise, it would be hacky to do a quick fix as it might introduce inconsistent behaviors where for example documents are retrieved twice or even not retrieved at all because of the rounding issues.

Given that, the ET for getting this fixed would be around a week to address the places that were clear during the timeboxed investigation and also other places that came up during the discussion with @weltenwort.

@ruflin wdyt? should we proceed with implementing the fix?

[ruflin]

++ on moving forward with this.

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)