Whether the monocular and IMU are the same clock？

[dongxuanlb]

Through firmware sync, we’re monitoring for drift and aligning the capture timestamps of the cameras, which are taken at the MIPI Start-of-Frame event. The Left/Right global shutter cameras are driven by the same clock, started by broadcast write on I2C, so no drift will happen over time, even when running freely without a sync. With the above functionality it would be also possible to configure FSIN as an output on one sensor, and an input to the other sensor. The RGB rolling shutter has a slight difference in clocking/frame-time, so when we detect a small drift, we’re modifying the frame time (number of lines) for the next frame by a small amount to compensate.

mentioned in this article

My question is whether the monocular camera and IMU are driven by the same clock？

in source：

1. RawIMUData.hpp
   /** Generation timestamp, synced to host time */
   Timestamp timestamp = {};

   /** Generation timestamp, direct device monotonic clock */
   Timestamp tsDevice = {};

2. RawImgFrame.hpp
   Timestamp ts = {}; // generation timestamp, synced to host time
   Timestamp tsDevice = {}; // generation timestamp, direct device monotonic clock

It seems they are driven by the same clock。

But，for my test， the mono frame’s timestamp and the imu frame's timestamp are not align.

[SzabolcsGergely]

My question is whether the monocular camera and IMU are driven by the same clock？

No. They aren't.
IMU timestamp that you see in IMU packet message is captured the moment interrupt is received from IMU.
Image timestamp is captured at the end of exposure/start of MIPI readout.
Only stereo cameras are HW synced

[dongxuanlb]

ok。So is there any good method or idea for how to synchronize between the two? The core problem is that I can't know the time offset between them through the sequence。

[SzabolcsGergely]

ok。So is there any good method or idea for how to synchronize between the two? The core problem is that I can't know the time offset between them through the sequence。

There's a confusion.
The IMU and cameras are not driven by the same clock, only mono cameras are driven by the same clock.
E.g. capture time is different for IMU and stereo cameras.
The timestamp captured and stored in messages are from the same free running counter. Therefore you can compare directly the timestamp from camera and timestamp from IMU since they are from the same source.

[dongxuanlb]

Maybe the word "synchronization" I described is wrong, "alignment" is what I want to express。

in my test(30fps mono, 500hz accel by callback), I found that imu seq was got from 12 or 13 normally. compared with the first mono frame's timestamp, it usually had a huge time offset(more than 100ms).

[2022-07-12 22:00:30.369] [info] got frame 1550.898, 0
[2022-07-12 22:00:30.397] [info] got frame 1550.931, 1
[2022-07-12 22:00:30.431] [info] got frame 1550.964, 2
[2022-07-12 22:00:30.466] [info] got frame 1550.998, 3
[2022-07-12 22:00:30.468] [info] got imu 1551.029, 12
[2022-07-12 22:00:30.471] [info] got imu 1551.031, 13
[2022-07-12 22:00:30.475] [info] got imu 1551.033, 14
[2022-07-12 22:00:30.475] [info] got imu 1551.035, 15
[2022-07-12 22:00:30.475] [info] got imu 1551.037, 16
[2022-07-12 22:00:30.476] [info] got imu 1551.039, 17
[2022-07-12 22:00:30.477] [info] got imu 1551.041, 18
[2022-07-12 22:00:30.479] [info] got imu 1551.043, 19
[2022-07-12 22:00:30.481] [info] got imu 1551.045, 20
[2022-07-12 22:00:30.498] [info] got frame 1551.031, 4
[2022-07-12 22:00:30.498] [info] got imu 1551.047, 21
[2022-07-12 22:00:30.500] [info] got imu 1551.049, 22
[2022-07-12 22:00:30.503] [info] got imu 1551.051, 23

as above，use timestamp(), the first imu frame's timestamp is 1551.005(1551.029 - 2ms x 12). if the clock is same, the offset is 1551.005 - 1550.898 = 0.107

[2022-07-12 22:48:17.379] [info] got frame 2357.374, 0
[2022-07-12 22:48:17.415] [info] got frame 2357.408, 1
[2022-07-12 22:48:17.445] [info] got frame 2357.441, 2
[2022-07-12 22:48:17.478] [info] got frame 2357.474, 3
[2022-07-12 22:48:17.487] [info] got imu 2357.513, 12
[2022-07-12 22:48:17.491] [info] got imu 2357.515, 13
[2022-07-12 22:48:17.492] [info] got imu 2357.517, 14
[2022-07-12 22:48:17.492] [info] got imu 2357.519, 15
[2022-07-12 22:48:17.509] [info] got imu 2357.521, 16
[2022-07-12 22:48:17.514] [info] got imu 2357.523, 17
[2022-07-12 22:48:17.514] [info] got frame 2357.507, 4
[2022-07-12 22:48:17.516] [info] got imu 2357.525, 18
[2022-07-12 22:48:17.517] [info] got imu 2357.527, 19

as above，use timestampDevice() the first imu frame's timestamp is 2357.489( 2357.513 - 2ms x 12) . if the clock is same, the offset is 2357.489 - 2357.374 = 0.115

[2022-07-12 22:53:49.420] [info] got frame 34.928, 0
[2022-07-12 22:53:49.451] [info] got frame 34.962, 1
[2022-07-12 22:53:49.481] [info] got frame 34.995, 2
[2022-07-12 22:53:49.517] [info] got frame 35.028, 3
[2022-07-12 22:53:49.518] [info] got imu 35.06, 12
[2022-07-12 22:53:49.521] [info] got imu 35.062, 13
[2022-07-12 22:53:49.522] [info] got imu 35.064, 14
[2022-07-12 22:53:49.523] [info] got imu 35.066, 15
[2022-07-12 22:53:49.525] [info] got imu 35.068, 16
[2022-07-12 22:53:49.528] [info] got imu 35.07, 17
[2022-07-12 22:53:49.529] [info] got imu 35.072, 18
[2022-07-12 22:53:49.531] [info] got imu 35.073, 19
[2022-07-12 22:53:49.533] [info] got imu 35.076, 20
[2022-07-12 22:53:49.552] [info] got imu 35.078, 21
[2022-07-12 22:53:49.553] [info] got frame 35.062, 4
[2022-07-12 22:53:49.554] [info] got imu 35.08, 22
[2022-07-12 22:53:49.555] [info] got imu 35.081, 23

as above，use timestampDevice() the first imu frame's timestamp is 35.036(35.06 - 2ms x 12) . if the clock is same, the offset is 35.036 - 34.928 = 0.108

so, it seems the offset is 100ms?

[SzabolcsGergely]

IMU and stereo cameras are running independently, there's no HW sync between them.
Sequence numbers are independent, timestamps are independent. In other words they are sampled independently.
The best way to align them is based on timestamp, which is taken from common source (free running counter with the same clock as source).
Offset is 100 ms, depending on how you measure it.
If you want to align IMU frames with camera frames, the best way to do it is based on device timestamp.
IMU frame X is aligned with frame Y if the timestamp of IMU frame is inside of the frame time interval for frame Y, e.g. [frame timestamp - 33.3 ms, frame timestamp]. Another, better way (for VIO, SLAM etc.) is aligning against the middle of exposure time. (PR to access exposure time is here: #191)

[dongxuanlb]

Ok！

free running counter with the same clock as source

This information ensures that even though they are sampled independently, their start times are the same, so alignment can be done. Thanks a lot for your answer.

[Erol444]

Hi @dongxuanlb ,
Here's a simple demo of IMU + frame syncing based on timestamps.
Thanks, Erik