What will realsense viewer do when I export the 3D mesh?

[CodeLHY]

• Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

  • Consider checking out SDK examples.
  • Have you looked in our documentations?
  • Is you question a frequently asked one?
  • Try searching our GitHub Issues (open and closed) for a similar issue.

• All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

---

Required Info
Camera Model	D435i
Firmware Version	05.13.00.50
Operating System & Version	Win 10
Kernel Version (Linux Only)	xxx
Platform	PC
SDK Version	2.50.0
Language	opencv/python
Segment	Robot

Issue Description

I recorded a .bag file, and tried to use pyrealsense2 to parse the .bag file to get the RGB and also the depth map. However, when I tried to visualize the depth map in the point cloud format, I found that the objects are not vertical but oblique, I am not sure what will cause this kind of problem. At the same time, I found that, the mesh exported through the GUI of RealSenseViewer looks good. So I wonder what will RealSenseViewer do when I click the "export" button.

To help you find out the problem, I paste the main code here.

"""
This file contains the tools to parse the .bag file, to get the original data.
Parser_Bag is a class to process a specific .bag file.
"""
from operator import index
from pdb import post_mortem
import pyrealsense2 as rs
import open3d as o3d
import numpy as np
import cv2
from copy import deepcopy
class Parser_Bag:
    def __init__(self, file_bag, post_proc = False, post_proc_dec=False, post_proc_spa=False, post_proc_tem=False, post_proc_hol=False) -> None:
        config   = rs.config()    # construct the config for realsense.
        pipeline = rs.pipeline()  # construct a pipeline
        config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 15)   # Enable a device stream explicitly, with selected stream parameters.
        config.enable_stream(rs.stream.color, 640, 480, rs.format.rgb8, 15)
        rs.config.enable_device_from_file(config, file_bag, repeat_playback=False)  # specify the device as the recorded device from a file.  https://github.com/IntelRealSense/librealsense/issues/3147#issuecomment-458145775
        profile = pipeline.start(config)  # start the stream of color and depth
        # set as play back
        device   = profile.get_device()
        playback = device.as_playback()
        playback.set_real_time(False)
        # set align
        align_to = rs.stream.color
        align    = rs.align(align_to)
        
        depth_sensor      = device.first_depth_sensor()  # get the first depth sensor
        self.depth_scale  = depth_sensor.get_depth_scale()  # get the depth scale of the sensor

        index_frame = 0
        self.post_proc = post_proc
        self.frames_rgb = []
        self.frames_dpt = []
        self._get_filters()
        self.intrinsic    = rs.video_stream_profile(pipeline.get_active_profile().get_stream(rs.stream.depth)).get_intrinsics()
        self.cx = self.intrinsic.ppx
        self.cy = self.intrinsic.ppy
        self.fx = self.intrinsic.fx
        self.fy = self.intrinsic.fy
        if post_proc and post_proc_dec:
            self.fx = self.fx // self.magnitude_dec
            self.fy = self.fy // self.magnitude_dec
            self.cx = self.cx // self.magnitude_dec
            self.cy = self.cy // self.magnitude_dec
        while True:
            try:
                frames = pipeline.wait_for_frames(timeout_ms=100)
                if frames.size() < 2:
                    # data is not ready
                    continue
            except(RuntimeError):
                print("totally: ", index_frame)
                pipeline.stop()
                break
            aligned_frames = align.process(frames)
            if self.post_proc:
                depth_frame = np.array(self.post_process(aligned_frames.get_depth_frame(), dec=post_proc_dec, spa=post_proc_spa, tem=post_proc_tem, hol=post_proc_hol).data)
            else:
                depth_frame = np.array(aligned_frames.get_depth_frame().data)
            color_frame = np.array(aligned_frames.get_color_frame().data)
            if depth_frame is None or color_frame is None:
                continue
            print(f"get depth frame_{index_frame}")
            self.frames_rgb.append(color_frame)
            self.frames_dpt.append(depth_frame)
            index_frame += 1
        # get the RGB/depth data for each frame.
        print("Depth Scale is: " , self.depth_scale)
        print("Camera Intrinsic Matrix is: ", self.intrinsic)
    def post_process(self, frame, dec=False, spa=False, tem=False, hol=False):
        if dec:
            frame = self.decimation_filter.process(frame)
        frame = self.depth_to_disparity.process(frame)
        if spa:
            frame = self.spatial_filter.process(frame)
        if tem:
            frame = self.temporal_filter.process(frame)
        frame = self.disparity_to_depth.process(frame)
        if hol:
            frame = self.hole_filling_filter.process(frame)
        return frame
    def _get_filters(self):
        # xxx_filter.process(frame)
        ### decimation filter
        self.decimation_filter = rs.decimation_filter()
        self.magnitude_dec     = 2 
        self.decimation_filter.set_option(rs.option.filter_magnitude, self.magnitude_dec)  # set the down sampling ratio.

        ### spatial filter
        self.spatial_filter    = rs.spatial_filter()  # Spatial Filter is a fast implementation of Domain-Transform Edge Preserving Smoothing
        self.spatial_filter.set_option(rs.option.filter_magnitude, 5)
        self.spatial_filter.set_option(rs.option.filter_smooth_alpha, 1)
        self.spatial_filter.set_option(rs.option.filter_smooth_delta, 50)
        self.spatial_filter.set_option(rs.option.holes_fill, 3)  # spatial offers basic hole filling algarithm.
        ### temporal filter
        self.temporal_filter = rs.temporal_filter()
        ### hole filling
        self.hole_filling_filter = rs.hole_filling_filter()
        self.depth_to_disparity = rs.disparity_transform(True)
        self.disparity_to_depth = rs.disparity_transform(False)
    def get_np_array(self, frame):
        return np.asanyarray(frame.data)
    def visualize_frame(self, index_frame):
        cld = self._dpt2cld(self.frames_dpt[index_frame])
        self.save_cloud(cld, f"./dataset_debug/cld_{index_frame}.ply")
        cv2.imwrite(f"./dataset_debug/RGB_{index_frame}.png", self.frames_rgb[index_frame])
    def _dpt2cld(self, dpt_map):
        # dpt_map: w h 3 np.array
        w, h = dpt_map.shape
        xmap = np.array([[j for i in range(h)] for j in range(w)])
        ymap = np.array([[i for i in range(h)] for j in range(w)])
        pt2 = dpt_map.flatten()[:, np.newaxis].astype(np.float32) * self.depth_scale
        pt0 = (ymap.flatten()[:, np.newaxis] - self.cx) * pt2 / self.fx
        pt1 = (xmap.flatten()[:, np.newaxis] - self.cy) * pt2 / self.fy
        cloud = np.concatenate((pt0, pt1, pt2), axis=1)
        return cloud
    def save_cloud(self, cloud, path_save, color=None):
        '''
        Description: save the point cloud

        Args:
            cloud: the point cloud. np.array. [N, 3]
            color: the color of each point. np.array. [N, 3], None--> all white

        '''
        print("WARNING! save_cloud")
        pc = o3d.geometry.PointCloud()
        if color is None:
            N, _  = cloud.shape
            color = np.array([[1,1,1]]).repeat(N, 0)
        pc.points = o3d.utility.Vector3dVector(cloud)
        pc.colors = o3d.utility.Vector3dVector(color)
        o3d.io.write_point_cloud(path_save, pc)
if __name__ == "__main__":
    parser_bag = Parser_Bag("../realsense_data/20220301_174251.bag", post_proc=False, post_proc_dec=False, post_proc_spa=False, post_proc_tem=False, post_proc_hol=True)
    from ipdb import set_trace; set_trace()
    parser_bag.visualize_frame(525)

[MartyG-RealSense]

Hi @CodeLHY It looks as though you are applying depth to color alignment with align_to before the post-processing filters are applied. Intel recommends performing alignment after post-processing to help to avoid distortion effects such as aliasing (jagged lines). Could you try moving alignment to a section of the script that comes after the filter list to see whether the image improves, please?

[CodeLHY]

In fact, the post-processing flag is set as False.

parser_bag = Parser_Bag("../realsense_data/20220301_174251.bag", post_proc=False, .......

Hi @CodeLHY It looks as though you are applying depth to color alignment with align_to before the post-processing filters are applied. Intel recommends performing alignment after post-processing to help to avoid distortion effects such as aliasing (jagged lines). Could you try moving alignment to a section of the script that comes after the filter list to see whether the image improves, please?

[MartyG-RealSense]

Thank you for the information.

Another difference between the RealSense Viewer and your own script may be that you are using the align_to method to perform depth to color alignment. The Viewer is likely using the alternative pc.calculate alignment method to map RGB data onto depth points directly, as demonstrated by the SDK's C++ 'rs-pointcloud' example program.

https://github.com/IntelRealSense/librealsense/tree/master/examples/pointcloud

[CodeLHY]

You are right, when I align the color to the depth map, the strange phenomenon disappears immediately.

align_to = rs.stream.depth
align    = rs.align(align_to)

By the way, where can I find the script that describes what Realsense Viewer does when I click "export" button exactly? Maybe this will help me to learn how to process the raw data captured by the RealSense Camera. Thank you for your prompt reply very much!

[MartyG-RealSense]

The Viewer is making use of the save_to_ply instruction, which provides a range of options for customizing ply export.

https://github.com/IntelRealSense/librealsense/blob/master/common/viewer.cpp#L111-L114

https://intelrealsense.github.io/librealsense/doxygen/classrs2_1_1save__to__ply.html

Using save_to_ply in Python has a history of being a problem though in regard to saving RGB color to ply. There is an alternative instruction called export_to_ply that has been shown to be able to save color to ply, though without normals. #6194 (comment) provides a Python example of saving such a color ply.

[MartyG-RealSense]

Hi @CodeLHY Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.