synchronization between imu measurement and stereo images

[SinnedHog]

I performed an imu-camera calibration using the open source library Kalibr (https://github.com/ethz-asl/kalibr) to find out the delay between the imu measurements and stereo images. I fixed the camera exposure time to 1ms and the result from the calibration indicated that the camera is 10ms slower than the imu timestamp. I have read in one of the github post (#516) that even though the imu and stereo cameras are not hardware sync but they are running on the same clock. Therefore, the delay should not be larger than half of the camera exposure time, which in this case is 0.5ms. Is the result from kalibr expected? My experience from using kalibr is that it has been reliably given me accurate estimate. Has anyone encountered the same issue before?

[Erol444]

Hi @SinnedHog ,
Here's a simple demo of IMU + frame syncing based on timestamps. I hope this helps.
Thanks, Erik

[SinnedHog]

Hi @Erol444,
I think is this not what I am looking for. What I meant is that I am still getting an time delay of 10ms even after aligning them using their timestamp. So the question is if the IMU time or mono images time is actually what it is? By the way, I did this calibration via ROS, not sure if it is the ROS wrapper (depthai-ros) problem. Do you have a script/program in depthai-python that allows me to log IMU measurements and mono images concurrently? I want to repeat the calibration using non ROS wrapper.

[Erol444]

Hi @SinnedHog ,
I don't think it's possible to hardware sync IMU yet, or to reduce drift from the camera stream via firmware (yet). So my naive approach would be to have a higher IMU frequency, so max delay would be lower (eg at 250Hz it would be max 2ms).
Thanks, Erik

[dongxuanlb]

Hi @SinnedHog

According to my recent experience, OAKD's left and right monocular and imu do use the same clock.
The reason for the feeling of delay in programming is mainly to do the "calibration" of the basic time. Here I refer to the imu example in the example directory.

 if (name == IMU_STREAM_NAME) {
    auto imuDataPtr = std::dynamic_pointer_cast<dai::IMUData>(data);
    auto imuPackets = imuDataPtr->packets;

    if(!firstTs) {
        for(auto& imuPacket : imuPackets) {
            _baseTime = std::min(imuPacket.acceleroMeter.timestamp.get(), imuPacket.gyroscope.timestamp.get());
            firstTs = true;
            break;
        }
    }

Then all other frame times can be obtained by the following helper functions：

double OAKD::getFrameTime(time_point<steady_clock, steady_clock::duration> time) {
    return duration_cast<milliseconds>(time - _baseTime).count() / 1000.;
}

For example in my code：

    auto rgbPtr = std::dynamic_pointer_cast<dai::ImgFrame>(data);
    auto seqId = rgbPtr->getSequenceNum();
    _syncFrameMutex.lock();
    SyncFramePtr& sync = _preallocatedSyncFramePtrs[seqId % preallocated_item_num];
    sync->rgbImg = rgbPtr->getFrame();
    sync->rgbSeqId = seqId;
    sync->rgbT = **getFrameTime(rgbPtr->getTimestamp())**;
    publishSyncFrame(sync);

[2022-11-01 19:49:12.074] [info] [OAK-D]Publish Sync Frame
[2022-11-01 19:49:12.078] [info] [OAK-D][LImg][0]0ms, [RImg][0]2.466ms
[2022-11-01 19:49:12.080] [info] [OAK-D][RGB ][0]0ms
[2022-11-01 19:49:12.080] [info] [OAK-D][LPts][0]0ms, [RPts][0]0ms
[2022-11-01 19:49:12.120] [info] [OAK-D]Publish Sync Frame
[2022-11-01 19:49:12.123] [info] [OAK-D][LImg][0]2.466ms, [RImg][0]2.466ms
[2022-11-01 19:49:12.126] [info] [OAK-D][RGB ][0]2.467ms
[2022-11-01 19:49:12.130] [info] [OAK-D][LPts][0]2.466ms, [RPts][0]2.466ms
[2022-11-01 19:49:12.151] [info] [OAK-D]Publish Sync Frame
[2022-11-01 19:49:12.153] [info] [OAK-D][LImg][1]2.5ms, [RImg][1]2.5ms
[2022-11-01 19:49:12.155] [info] [OAK-D][RGB ][1]2.501ms
[2022-11-01 19:49:12.155] [info] [OAK-D][LPts][1]2.5ms, [RPts][1]2.5ms
[2022-11-01 19:49:12.192] [info] [OAK-D]Publish Sync Frame
[2022-11-01 19:49:12.196] [info] [OAK-D][LImg][2]2.533ms, [RImg][2]2.533ms
[2022-11-01 19:49:12.198] [info] [OAK-D][RGB ][2]2.534ms
[2022-11-01 19:49:12.200] [info] [OAK-D][LPts][2]2.533ms, [RPts][2]2.533ms

2022-11-01 19:53:39.370] [info] [imu]-A[158]4.822ms, G[139]4.824ms
[2022-11-01 19:53:39.370] [info] [imu]-A[160]4.826ms, G[140]4.826ms
[2022-11-01 19:53:39.370] [info] [imu]-A[161]4.828ms, G[141]4.828ms
[2022-11-01 19:53:39.370] [info] [imu]-A[162]4.83ms, G[142]4.831ms
[2022-11-01 19:53:39.370] [info] [imu]-A[163]4.832ms, G[143]4.834ms
[2022-11-01 19:53:39.389] [info] [imu]-A[165]4.836ms, G[144]4.836ms
[2022-11-01 19:53:39.389] [info] [imu]-A[166]4.838ms, G[145]4.838ms
[2022-11-01 19:53:39.389] [info] [imu]-A[167]4.84ms, G[146]4.841ms
[2022-11-01 19:53:39.389] [info] [imu]-A[168]4.842ms, G[147]4.843ms
[2022-11-01 19:53:39.389] [info] [imu]-A[170]4.846ms, G[148]4.846ms
[2022-11-01 19:53:39.390] [info] [OAK-D]Publish Sync Frame
[2022-11-01 19:53:39.390] [info] [OAK-D][LImg][148]4.832ms, [RImg][148]4.832ms
[2022-11-01 19:53:39.390] [info] [OAK-D][RGB ][148]4.832ms
[2022-11-01 19:53:39.390] [info] [OAK-D][LPts][148]4.832ms, [RPts][148]4.832ms
[2022-11-01 19:53:39.391] [info] [imu]-A[171]4.848ms, G[149]4.848ms
[2022-11-01 19:53:39.391] [info] [imu]-A[172]4.85ms, G[150]4.851ms
[2022-11-01 19:53:39.391] [info] [imu]-A[173]4.851ms, G[151]4.854ms
[2022-11-01 19:53:39.391] [info] [imu]-A[175]4.856ms, G[152]4.856ms
[2022-11-01 19:53:39.392] [info] [imu]-A[176]4.858ms, G[153]4.858ms
[2022-11-01 19:53:39.394] [info] [imu]-A[177]4.859ms, G[154]4.861ms
[2022-11-01 19:53:39.394] [info] [imu]-A[179]4.864ms, G[155]4.863ms
[2022-11-01 19:53:39.394] [info] [imu]-A[180]4.865ms, G[156]4.866ms
[2022-11-01 19:53:39.394] [info] [imu]-A[181]4.867ms, G[157]4.868ms
[2022-11-01 19:53:39.394] [info] [imu]-A[182]4.869ms, G[158]4.871ms
[2022-11-01 19:53:39.396] [info] [imu]-A[184]4.874ms, G[159]4.873ms
[2022-11-01 19:53:39.396] [info] [imu]-A[185]4.875ms, G[160]4.876ms
[2022-11-01 19:53:39.396] [info] [imu]-A[186]4.877ms, G[161]4.878ms
[2022-11-01 19:53:39.396] [info] [imu]-A[187]4.879ms, G[162]4.881ms
[2022-11-01 19:53:39.396] [info] [imu]-A[189]4.883ms, G[163]4.883ms
[2022-11-01 19:53:39.428] [info] [imu]-A[190]4.885ms, G[164]4.886ms
[2022-11-01 19:53:39.428] [info] [imu]-A[191]4.887ms, G[165]4.888ms
[2022-11-01 19:53:39.428] [info] [imu]-A[193]4.891ms, G[166]4.891ms
[2022-11-01 19:53:39.428] [info] [imu]-A[194]4.893ms, G[167]4.893ms
[2022-11-01 19:53:39.428] [info] [imu]-A[195]4.895ms, G[168]4.896ms
[2022-11-01 19:53:39.428] [info] [OAK-D]Publish Sync Frame
[2022-11-01 19:53:39.428] [info] [OAK-D][LImg][149]4.865ms, [RImg][149]4.865ms
[2022-11-01 19:53:39.428] [info] [OAK-D][RGB ][149]4.865ms
[2022-11-01 19:53:39.428] [info] [OAK-D][LPts][149]4.865ms, [RPts][149]4.865ms

[SinnedHog]

Hi @dongxuanlb, thanks for your response. I understand that according to the explanation given in #516 (which is asked by you), both imu timestamp and mono image timestamp are gotten from the same clock. in that case, by just comparing their timestamp, we can align them together. However, my question is really if there is any delay when assigning the timestamp to the imu measuremnt and the mono image measurement. Have you tried to perform a camera-imu calibration such as using the Kalibr (https://github.com/ethz-asl/kalibr) to find out whether the imu and camera are truly synchronized?
After performing the camera-imu calibration, the result shows that there is a 7-10ms delay between these two sources which shouldn't be the case since my IMU rate is around 180Hz, therefore the delay should be less than 1.0/180/2 = 3ms. Therefore, I was wondering could there be any delay in assigning the timestamp to either imu or mono image when they are read in by the board.

[saching13]

@SinnedHog there is no delay in assigning the timestamp from our debugging. But it looks like there is some delay is getting the image frames out of USB.

In the above image the plots are plotted where X axis is the time at which messages are published. And Y axis represents the header timestamp. Form this you can see that the header timestamp of image comes later relative to the IMU which is batched for better throughput.
But if we look at it form the header.timestamp being the X axis. they overlap as shown in the image below. Which means that there if you create a queue of IMU messages you can look-up for the closest IMU sample to the current image received for better sync here.

Thoughts ?