Point Cloud Looks Distorted

[andrearosasco]

---

Required Info
Camera Model	D435i
Firmware Version	2.50.0.3785
Operating System & Version	Win 10
Platform	PC
SDK Version	pyrealsense2 2.50.0.3812
Language	python
Segment	Robot

Issue Description

I'm trying to recreate the pointcloud of the environment starting from the depth image using the python API. While the pointcloud visualized with RealSense Viewer looks fine, the one I compute using python APIs and open3d is somehow distorted.

Specifically I noticed that putting a box in front of the camera I get a pointcloud where the angle between the front face (the one towards the camera) and the upper face is more than 90 degree. I think it might have to do with the intrinsic parameters but I get them through profile.get_stream(rs.stream.depth).as_video_stream_profile().get_intrinsics() so they should be fine.

Here is some code to reproduce the issue:

import json
from pathlib import Path

import pyrealsense2 as rs
import numpy as np
import open3d as o3d
from open3d.cpu.pybind.visualization import draw_geometries


class RealSense:
    def __init__(self, width, heigth):
        self.pipeline = rs.pipeline()

        config = rs.config()
        config.enable_stream(rs.stream.depth, width, heigth, rs.format.z16, 30)
        config.enable_stream(rs.stream.color, width, heigth, rs.format.bgr8, 30)
        self.profile = self.pipeline.start(config)

        HIGH_ACCURACY = 3
        HIGH_DENSITY = 4
        MEDIUM_DENSITY = 5
        self.profile.get_device().sensors[0].set_option(rs.option.visual_preset, HIGH_DENSITY)

        self.align = rs.align(rs.stream.color)

    def intrinsics(self):
        return self.profile.get_stream(rs.stream.depth).as_video_stream_profile().get_intrinsics()

    def read(self):
        frames = self.pipeline.wait_for_frames()
        aligned_frames = self.align.process(frames)

        depth_frame = aligned_frames.get_depth_frame()  # aligned_depth_frame is a 640x480 depth image
        color_frame = aligned_frames.get_color_frame()

        depth_image = np.asanyarray(depth_frame.get_data())
        color_image = np.asanyarray(color_frame.get_data())

        return color_image, depth_image

    def pointcloud(self, depth_image):
        depth_image = o3d.geometry.Image(depth_image)

        intrinsics = self.intrinsics()
        camera = o3d.camera.PinholeCameraIntrinsic(intrinsics.width, intrinsics.height, intrinsics.fx,
                                                   intrinsics.fy, intrinsics.ppx, intrinsics.ppy)

        pcd = o3d.geometry.PointCloud.create_from_depth_image(depth_image, camera)
        pcd.transform([[1, 0, 0, 0],
                       [0, -1, 0, 0],
                       [0, 0, -1, 0],
                       [0, 0, 0, 1]])

        return pcd


if __name__ == '__main__':
    camera = RealSense(640, 480)

    rgd, depth = camera.read()
    pc = camera.pointcloud(depth)
    
    draw_geometries([pc])

And a lateral image of the box I was talking about:

Thank you in advance for your help!

[MartyG-RealSense]

Hi @andrearosasco There was a past Python case in #6857 where a RealSense user was experiencing a 90 degree angle when generating a point cloud of a box.

The eventual solution that worked for them in their particular case was to create a custom json camera configuration file, which they shared the contents of in #6857 (comment)

[andrearosasco]

Thanks, I'll try that! It's strange though... In my code I tried some of the preset configurations so if it's a configuration issue it should be present in the viewer as well. Can it be that the viewer is doing some kind of distortion correction that is not automatically done by the API?

Also, do you know how can I load the custom JSON from #6857? I can't find any function to load a custom configuration. Do I have to do it manually by iterating the dictionary and calling set_option()

I speculate that the issue I'm experiencing is actually very common if you compute the point cloud manually starting from the depth frame. I was probably experiencing it for months but I just noticed it when I looked real close to the box point cloud.

EDIT: actually it looks like the guy from #6857 was experiencing this problem on the Viewer while in my case the RealSense Viewer works just fine

[andrearosasco]

Ok, I just found this #5506 and it solved it for me. I just stopped aligning the color and depth frame and now everything works fine.

[MartyG-RealSense]

The ShortRangePreset json that the RealSense user in that case adapted is not included in the Viewer by default on 400 Series cameras. It is also not the same as the similarly named Short Range preset that the Viewer offers owners of the L515 camera model. It was instead obtained from the Visual Preset documentation page in the section titled 'Additional Presets not in the Viewer'.

https://dev.intelrealsense.com/docs/d400-series-visual-presets#section-preset-table

The user in that case experienced problems in both the Viewer and their Python application, as they said "When streaming depth data with the realsense viewer (or my own app) the walls of the box on the ground floor are hanging towards the outside of the box".

Anyway, it's great to hear that you found a solution. Thanks very much for the update!

[MartyG-RealSense]

Case closed due to solution achieved and no further comments received.