Mapping coordinates from depth to colour results in outside-of-colour-image-boundaries coordinates.

[vladdders]

Required Info
Camera Model	D415
Firmware Version	05.08.15.00
Operating System & Version	Linux (Ubuntu 16)
Kernel Version (Linux Only)	4.13.0-45-generic
Platform	UP board with Intel Atom x5-Z8350 CPU @ 1.44Ghz x 4
SDK Version	2.11.0
Language	python

Issue Description

Hi,

I am trying to use this method of mapping the pixels from the depth channel to colour channel. I am using the Python API and apparently the rs.align(align_to) method is very CPU intensive and it dramatically slows down my application. Thus, I was thinking maybe mapping depth pixels to colour pixels would make things run a little faster. However, I ran into the following problem: when I try to map depth pixel [200, 200] to its corresponding colour pixel I get returned [15294.3466796875, -113.68994140625]. I doubt this is correct since my colour channel resolution is 1280x720 and these coordinates are clearly out of [0, 1280], [0, 720]. This is what gets printed:

Depth intrinsics: width: 1280, height: 720, ppx: 644.664, ppy: 406.596, fx: 965.563, fy: 965.563, model: Brown Conrady, coeffs: [0, 0, 0, 0, 0]

 Color intrinsics: width: 640, height: 480, ppx: 330.929, ppy: 233.255, fx: 610.102, fy: 609.615, model: Brown Conrady, coeffs: [0, 0, 0, 0, 0]

 Depth to color extrinsics: rotation: [0.99999, 0.00428028, 0.0010846, -0.00427981, 0.999991, -0.000430169, -0.00108643, 0.000425523, 0.999999]
translation: [0.0151529, -0.00012542, -0.000400548]

	 depth_pixel: [200, 200]

	 color_pixel: [15294.3466796875, -113.68994140625]

Are there any other transformations I need to carry out? Any help is very much appreciated.

[dorodnic]

Could you please share the exact code snippet you are using?
In general not every pixel on the depth image will fall inside the RGB image, due to sensor displacement and differences in FOV, but for the D415 the effect should not be significant, and certainly not that extreme.

[vladdders]

So, my code is:

import pyrealsense2 as rs

config = rs.config()
config.enable_stream(rs.stream.depth, 1280, 720, rs.format.z16, 30)
config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)

pipeline = rs.pipeline()

pipe_profile = pipeline.start(config)

frames = pipeline.wait_for_frames()
depth_frame = frames.get_depth_frame()
color_frame = frames.get_color_frame()
# Intrinsics & Extrinsics
depth_intrin = depth_frame.profile.as_video_stream_profile().intrinsics
color_intrin = color_frame.profile.as_video_stream_profile().intrinsics
depth_to_color_extrin = depth_frame.profile.get_extrinsics_to(color_frame.profile)
color_to_depth_extrin = color_frame.profile.get_extrinsics_to(depth_frame.profile)

print "\n Depth intrinsics: " + str(depth_intrin)
print "\n Color intrinsics: " + str(color_intrin)
print "\n Depth to color extrinsics: " + str(depth_to_color_extrin)

# Depth scale - units of the values inside a depth frame, i.e how to convert the value to units of 1 meter
depth_sensor = pipe_profile.get_device().first_depth_sensor()
depth_scale = depth_sensor.get_depth_scale()
print "\n\t depth_scale: " + str(depth_scale)

depth_pixel = [200, 200]   # Random pixel
print "\n\t depth_pixel: " + str(depth_pixel)
depth_point = rs.rs2_deproject_pixel_to_point(depth_intrin, depth_pixel, depth_scale)
color_point = rs.rs2_transform_point_to_point(depth_to_color_extrin, depth_point)
color_pixel = rs.rs2_project_point_to_pixel(color_intrin, color_point)
print "\n\t color_pixel: " + str(color_pixel)

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
@vladsb94
Please change last parameter "depth_scale" in line of

depth_point = rs.rs2_deproject_pixel_to_point(depth_intrin, depth_pixel, depth_scale)

to the real distance of that specific point, I post my solution coded in C++, please find the relative api in python.
​rs2_deproject_pixel_to_point(depth_point, &d_intrin, depth_pixel, dframe.as<depth_frame>().get_distance(200,200));

last parameter requires the real depth data in that specific pixel instead of the depth scale.