Global timestamps wrong after long use

[mikg13]

Required Info
Camera Model	D435i
Firmware Version	05.11.06.250
Operating System & Version	Ubuntu 18.04
Kernel Version (Linux Only)	4.18.0-25-generic
Platform	PC
SDK Version	2.24.0
Language	C++
Segment	Others

Issue Description

After a long period of time continually streaming from my D435i (anywhere between 10 minutes and 5 hours), I find that the timestamps begin to be largely out of sync with the system clock.

The D435i uses a global time domain, which as far as I understand should sync time periodically with the device it's connected to. I notice that it does attempt to synchronize by adjusting frame timestamps by a few milliseconds, which I'm assuming is normal, occasionally during normal operation.

After anywhere between 10 minutes and 5 hours (typically around 2 hours) the timestamps are wildly different and jump 50 years into the future 🤖

I've modified the rs_callback example so you can reproduce what I've found, please let me know if I've done anything fundamentally wrong in that modified example.
rs-callback.txt

The log I produce from that example is as follows:
callback_rs3.log

I'm aware there's already an issue of global timestamps being erratic within the first 15 seconds (Global Timestamp: first 15 seconds of frames timestamps are unstable (DSO-12942)), so I'm ignoring any wrong timestamps within that time.

Let me know if there's any way I can workaround it in the meantime, maybe there's something I can do with my system clock. Perhaps an NTP server could interfere?

[RealSenseCustomerSupport]

---

Hi @mikg13,

The issue is currently under investigation.Updates will be posted as soon as investigation findings become available.

Thank you!

[JeremyBYU]

I have this same issue. Note the frame timestamp discrepancy from time of arrival. Funny enough the first few seconds show a good time (157.....blah) then it jumps to (314....blah).

Setup: D435i, Ubuntu 19.10 (kernel 5.3), Firmware latest, SDK 2.30 and 2.29, C++ and realsense. Patched and upatched kernel.

EDIT:

Just noticed that while I am watching that it switched from 157 timestamp to a 314 timestamp when the hardware FPS counter drops to 0.02. This periodically occurs, however no errors in logs about this (just standard UVC meta data stuff)

[JeremyBYU]

So I have hunted down the issue a bit and have a better understanding.

When using global timestamps the timestamp returned is the device hardware timestamp synchronized with the host time. In order to do this a linear regression model is created that takes samples. These samples are (x,y) pairs, where x is the device time and y is the host time. A linear regression is performed on the samples to generate a baisc line mode (y=mx+b). This sampling process takes place in:

librealsense/src/global_timestamp_reader.cpp

Line 144 in 2decb32

double sample_hw_time = _device->get_device_time_ms();

Notice the get_device_time_ms(). This function is an api call that seems to actually send a command (using a class called hwmon) to return back the internal clock of the device. This is always successful for my case.

However this call is used to gather data to generate the model. To use the model an input timestamp from the frame of interest is sent. This is done here:

librealsense/src/global_timestamp_reader.cpp

Line 216 in 2decb32

double frame_time = _device_timestamp_reader->get_frame_timestamp(mode, fo);

This is where the issue comes in. My accelerator and gyro for my patched kernel seem to be working and give back good HID timestamps. These frame timestamps are very similar to the ones the hwmon api call gives. I have checked this with lots of logging. However my RGB and depth camera seem to have issues and I get the errror:
13/11 15:22:16,663 WARNING [139792792680192] (ds5-timestamp.cpp:64) UVC metadata payloads not available. Please refer to the installation chapter for details.

When this happens a backup reader is used:

librealsense/src/ds5/ds5-timestamp.cpp

Line 67 in 2decb32

return _backup_timestamp_reader->get_frame_timestamp(mode, fo);

Turns out this backup reader actually just gives the host time. But its expecting the device time! The host time and device time are very different from eachother. The result is the host frame timestamp is sent to the regressed model (trained on input device timestamps for x) leading to incredibly bad predictions for y, a synchonized global time.

So two things come to my mind. Realsense knows that the video timestamp is unavailable and is using the backup reader (host time). If it knows that it should not allow global time synchronization for that camera. Second I need to figure out how to fix the driver so that UVC metadata is working...

[JeremyBYU]

Here is some further information. The UVC Metadata warning only occurs after about 5-10 seconds for me. In the beginning reading the frame timestamp is not a problem and all is working (including global timestamps). Then an error occurs and the backup reader (giving host time) is switched to causing the issue.

As for that uvc driver I'm using, I have tried the following:

1. Raw unpatched kernel (5.3.0.20, Ubuntu 18.04).
2. Patched kernel from manually using realsense patches for bionic 18.04 (has suffix hwe). I believe these are technically for 5.0 or 5.2
3. Patches from another user that is for kernel 5.3.7. See Kernel patches will need UVC_INFO_QUIRK #5147

In all cases the patches are applied successfully, compiled successfully, and loaded successfully. They all suffer from the same problem though.

Not sure how to proceed. Note that I do not have any of these issues on Windows 10.

[doronhi]

The problem should be fixed with firmware 5.12.1.0. Could you please check it out?

[JeremyBYU]

This firmware update seemed to fix the problem for me. The timestamp looked correct for ~5 minutes. I'll let you know if any other issues surface in more testing.

Update - It seems I'm still having some issues. The UVC Meta Data error is still showing up -

07/01 11:10:58,158 WARNING [140089430611712] (ds5-timestamp.cpp:76) UVC metadata payloads not available. Please refer to the installation chapter for details.

However the frame timestamps no longer jumps and go haywire and stay steady (157... instead of 314...). However the timestamps domain still shows Global Domain which I highly doubt. These mentioned issues are only with RGB and Depth Camera sensors. Accel and Gyro dont have any issues I believe.

And now a new issue appears. If the realsense camera (d435i) is plugged in before I start realsense-viewer then an error message appears in realsense-viewer informing that the motion module is forced paused. It does not work when activated (toggle switch in realsense-viewer). Everything is fine with the motion sensors if I plug in the camera after realsense-viewer is started.

I have switched back to RSUSB backend. It seems to be stable and working well enough for me. I will try again on my main desktop once you have kernel drivers for 5.3 that are tested and working.

[RealSenseCustomerSupport]

---

Hi @JeremyBYU,

We wanted to make sure that the matter was resolved. Please let us know whether or not you continue to experience timestamp issue.

Thank you!

[JeremyBYU]

This specific timestamp issue (jumping to a bizarre timestamp) seems to be resolved. However the new firmware seems to be causing some other issues for me (see my comment above about the motion module being frozen). I'm actually still seeing this error sometimes even using the RSUSB backend. However that's a separate issue an I may file a bug after collecting more data.

[mikg13]

I haven't been able to verify that the timestamp issue has been resolved by this fix, as there seem to be new issues causing problems now with SDK 2.31. However I previously worked around the timestamp issue by using either the 4.15.0 or 5.0.0 kernel on ubuntu 18.04 with SDK 2.24.

After updating to SDK 2.31 and firmware 05.12.01.00 with my D435i, I get the following issues - see this linked video - when setting 'Global Time Enabled'. This will probably need to be settled in a separate thread, but for now I am not concerned about timestamp issues so I am ok with this branch being closed.

[RealSenseCustomerSupport]

---

@mikg13,@JeremyBYU -

Thank you for the feedback. If you don't have further questions regarding the original issue, let's close this thread and discuss each additional topic (that isn't related to the original one) in its own thread as a separate issue.

[matlabbe]

@mikg13 I reproduced the problem of your video on kernel 5.3.0-26 and 5.0.0-37. Only on 4.15.0-74 the timestamps are not flickering and increasing correctly. Tested with:

realsense-viewer v2.31.0 (binaries)
D435i (Firmware 05.12.01.00)