Getting accurate depth value after converting both color and depth images to OpenCV matrices

[prantoran]

• Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

  • Consider checking out SDK examples.
  • Have you looked in our documentations?
  • Is you question a frequently asked one?
  • Try searching our GitHub Issues (open and closed) for a similar issue.

• All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

---

Required Info
Camera Model	D435i
Firmware Version	-
Operating System & Version	Linux (Ubuntu 20.04 LTS)
Kernel Version (Linux Only)	5.4.0-53-generic
Platform	PC
SDK Version	~ 2.40.0
Language	C/C++
Segment	VR/AR

Issue Description

I am trying to integrate depth input into my project, which mainly uses OpenCV matrices to represent images. I was able to align the colour image with the depth image and convert both of them to OpenCV matrices. However, the corresponding depth values in the depth image for the queried positions in the aligned colour image are not stable (changes dramatically from frame to frame). Also, there is noise in the colour image (due to alignment maybe).

I am looking for the best way to align both the depth and colour images while minimizing noise, convert the colour image to an OpenCV matrix and get reliable depth values from the depth image for the corresponding positions in the colour image.

[MartyG-RealSense]

Hi @prantoran As a starting point in investigating your case, may I ask whether your cv-generated image resembles the depth-color aligned one in the link below, please?

#7852

[prantoran]

My aligned colour image is not mosaic but noisy and grainy. But the main problem for me to get depth values of pixel locations in the colour image as accurately as possible. I checked the depth map by showing grayscale and by applying rgb colour filter. The grayscale version seemed more unstable and flickering than the colour version of the depth map.

Thanks for linking the related issue, I will try out and inform if there is any improvement

Another thing, is the coordinate system of RealSense's Depth_Frame same as OpenCV's Matrices, as in the top-left position is (0, 0)?

[MartyG-RealSense]

I look forward to hearing about your test results.

In the RealSense coordinate system, the origin point is the center of the left IR sensor. Information about how values are positive or minus depending on whether they are left / right or above / below the center-point of the left IR sensor can be found in the link below.

#7279 (comment)

[prantoran]

@MartyG-RealSense I was able to run the rs2::depth_frame.get_distance(x, y) (with (0, 0) representing the top-left position) on the depth image for the corresponding (x, y) location in the colour image in OpenCV format (with (0, 0) representing the top-left position) but the values are unstable unresponsive to depth changes of objects within the field of view. I followed the documentation about projection.

I did apply align with respect to the colour image and I figure that this caused down-sampling of the depth frame. Is there a way to project positions of the colour frame to unaligned depth frame?

[MartyG-RealSense]

If depth to color alignment of the entire image is not ideal for you then the instruction rs2_project_color_pixel_to_depth_pixel may be helpful.

#7179

[MartyG-RealSense]

Hi @prantoran Do you require further assistance with this case, please? Thanks!

[prantoran]

I could not project colour pixel to depth pixel but continuing to try out the rs2::depth_frame.get_distance(x, y). The depth distance is proportional when the finger is in the centre of the camera (around 0.39 m) but breaks down to high values (ceiled at 2.0 m) when the finger goes near the corners of the camera.

depdemo.mp4

[MartyG-RealSense]

I have seen a couple of previous cases where the measurement is accurate at the center of the image but drifts for the coordinates at the side of the image when trying to obtain coordinates with get_distance. The links for these are below.

#6749

#7395

[MartyG-RealSense]

Hi @prantoran Do you require further assistance with this case, please? Thanks!

[prantoran]

@MartyG-RealSense Thank you so much for the help and linking relevant previous issues. I will go through them and closing this issue.

[MartyG-RealSense]

Thanks very much for the update!

[yangminwei95]

I look forward to hearing about your test results.

In the RealSense coordinate system, the origin point is the center of the left IR sensor. Information about how values are positive or minus depending on whether they are left / right or above / below the center-point of the left IR sensor can be found in the link below.

#7279 (comment)

Which is the left IR? Is it the one closer to the color camera or the one farther away?

[yangminwei95]

Is the picture correct？

[MartyG-RealSense]

Yes, the left IR is next to the color camera on the right side of the camera when looking at the camera from the front. The data sheet document for the 400 Series cameras provides the following description.

---

From the perspective of the stereo camera looking out at the world, the left imager is on the left side of the camera module.
Thus, when the user is facing the D4 camera, the left imager is actually on the right side of the camera module.

[yangminwei95]

Thanks for your reply. I get it.