How to create pointcloud from depth image correctly in open3d?

[TAW-V]

Required Info
Camera Model	d415
Firmware Version	5.15.0.2
Operating System & Version	ubuntu 22.04
Kernel Version (Linux Only)	5.15.0-78
Platform	PC
SDK Version	2.54.1.0
Language	python

Issue Description

I'm trying to create a pointcloud from a depth map in open3d using the camera intrinsics.
(I want to do this in open3d because I want to apply custom post-processing filters on the depth map, and I think its not possible to reconstruct a pyrealsense depth frame from a depth map in python.)

The pointcloud created using pyrealsense and the pointcloud created from a depth map in open3d differ a lot. The one created using pyrealsense looks correct. The open3d pointcloud is much narrower in x and y, but stretched in z.
(I applied filters to the depth map to smooth it to make the difference easier to see):

pc = rs.pointcloud()
points = pc.calculate(depth_frame)
v = points.get_vertices()
verts = np.asanyarray(v).view(np.float32).reshape(-1, 3)  # xyz

pcd = o3d.geometry.PointCloud()
pcd.points = o3d.utility.Vector3dVector(verts)

# Get stream profile and camera intrinsics
profile = pipeline.get_active_profile()
depth_profile = rs.video_stream_profile(profile.get_stream(rs.stream.depth))
depth_intrinsics = depth_profile.get_intrinsics()

# Convert the depth frame to a numpy array
depth_scale = device.query_sensors()[0].get_option(rs.option.depth_units)
depth_image_np = np.asanyarray(depth_frame.get_data()) * depth_scale

# Convert the numpy array to an open3d depth image
depth_image_o3d = o3d.geometry.Image(depth_image_np.astype(np.float32))

# Create an open3d camera intrinsics object from pyrealsense2 intrinsics
o3d_camera_intrinsic = o3d.camera.PinholeCameraIntrinsic(depth_intrinsics.width, depth_intrinsics.height, 
                                                          depth_intrinsics.fx, depth_intrinsics.fy, 
                                                          depth_intrinsics.ppx, depth_intrinsics.ppy)

# Create the point cloud from the open3d depth image
pcd2 = o3d.geometry.PointCloud.create_from_depth_image(depth_image_o3d, o3d_camera_intrinsic)

vis = o3d.visualization.Visualizer()
vis.create_window(window_name='pointcloud', width=3000, height=2000)
vis.add_geometry(pcd)
vis.add_geometry(pcd2)
vis.run()
vis.destroy_window()
pipeline.stop()

The intrinsics of the camera look correct:

print(depth_intrinsics.width, depth_intrinsics.height, 
           depth_intrinsics.fx, depth_intrinsics.fy, 
           depth_intrinsics.ppx, depth_intrinsics.ppy)
1280 720 889.8737182617188 889.8737182617188 633.4789428710938 344.9337158203125

What could be the problem?

[MartyG-RealSense]

Hi @thve25 The RealSense SDK has a range of post-processing filters that can be applied in pyrealsense2 by adding code to the script. Would any of these filters be able to provide the results that you were aiming for with custom post-processing in your Open3D script and therefore avoid the need to use Open3D?

https://dev.intelrealsense.com/docs/post-processing-filters

If you would prefer to use Open3D then the code shared by a RealSense user at isl-org/Open3D#473 (comment) for generating a real-time RealSense pointcloud in Open3D may be a helpful reference that you can compare to your own script.

[TAW-V]

Hi @thve25 The RealSense SDK has a range of post-processing filters that can be applied in pyrealsense2 by adding code to the script. Would any of these filters be able to provide the results that you were aiming for with custom post-processing in your Open3D script and therefore avoid the need to use Open3D?

Yes, that would be ideal. However, for my case I want to remove 20% of pixels of the depth map and replace them by the average of the surrounding pixels. What I have found this far, is that its not possible to modify pixels in the pyrealsense depth frame.

https://dev.intelrealsense.com/docs/post-processing-filters

If you would prefer to use Open3D then the code shared by a RealSense user at isl-org/Open3D#473 (comment) for generating a real-time RealSense pointcloud in Open3D may be a helpful reference that you can compare to your own script.

Yes, in isl-org/Open3D#473 (comment) he does something similar as me, but the my pointclouds do not match for some reason.

[TAW-V]

I've solved it. I used the wrong intrinsics.

This is correct:

frames = pipeline.wait_for_frames()
frames = align.process(frames)
profile = frames.get_profile()
intrinsics = profile.as_video_stream_profile().get_intrinsics()

This is not correct:

depth_profile = rs.video_stream_profile(profile.get_stream(rs.stream.depth))
depth_intrinsics = depth_profile.get_intrinsics()

However, modifying the depth frame directly would be ideal..

[MartyG-RealSense]

It's great to hear that you achieved a solution. Thanks very much for the update!

If depth is aligned to color then the color intrinsics should be used, as in the aligned image the color sensor's center-line becomes the origin coordinate for depth.

[MartyG-RealSense]

Case closed due to solution achieved and no further comments received.

[sanjaiiv04]

@MartyG-RealSense Hey I am trying to get point cloud of a specific region from depth map. To isolate an object in the depth map, I am using a segmentation model and then get the mask coordinates and overlay on the depth map. This is the result of that process:

Is it possible to save this cropped depth map as a point cloud and visualize it in open3d? If not, is there any other way to save it as a point cloud?

[MartyG-RealSense]

Hi @sanjaiiv04 Are you using C++ or Python programming language to generate your depth map, please?

[sanjaiiv04]

@MartyG-RealSense I am using Python here.

[MartyG-RealSense]

I am not expert on Open3D programming, unfortunately. But there is an Open3D pyrealsense2 pointcloud script at isl-org/Open3D#473 (comment) that you could potentially adapt to add your cropping to it.