Getting the 'vertices' of my D435 Depth Camera after aligning the depth frame to colour frame MATLAB

[hshorty]

I have a problem getting the 'vertices' of my D435 Depth Camera after aligning the depth frame to colour frame. The error I get is as follow. (Sometimes the code works fine). I do change the resolution of my camera. As I noted 'pnts.get_vertices()' works fine when I don't align the frames.

**Error using librealsense_mex
null pointer passed for argument "frame"

Error in realsense.points/get_vertices (line 13)
vertices = realsense.librealsense_mex('rs2::points', 'get_vertices', this.objectHandle);

Error in RealSencepredic_track_redObject_V3_30_06_2020 (line 140)
vertices = pnts.get_vertices(); % HS [x y z]**

Does anyone have an idea on solving this issue?

[MartyG-RealSense]

Hi @hshorty The link below has a MATLAB script contributed by a RealSense user for aligning depth to color, generating the point cloud and getting the vertices with vertices = pnts.get_vertices. It may be a useful reference for comparing to your own script's approach.

https://community.intel.com/t5/Items-with-no-label/MATLAB-RGB-point-cloud/m-p/584724#M10956

[hshorty]

Hi @MartyG-RealSense ,

Initially, I stared my code with code in the link you provided. However, it also had the same issue with MATLAB.

[hshorty]

Hi @MartyG-RealSense,

Thanks for the reply. However, the script you send me doesn't align the depth frame to colour frame (in this way you will not find any issue).

depth = fs.get_depth_frame();
%depth_scale = depth_sensor.get_depth_scale();
points = pointcloud.calculate(depth);
%921600x3 double xyz values
vertices = points.get_vertices();

I guess there is an issue in pointcloud.m-->realsense.librealsense_mex. Actually the real issue is not in 'get_vertices()'.

This is my code:

pipe = realsense.pipeline();

cfg = realsense.config();
cfg.enable_stream(realsense.stream.color,-1, 1280, 720, realsense.format.rgb8, 30);
cfg.enable_stream(realsense.stream.depth,-1, 1280, 720, realsense.format.z16, 30);

% Make Colorizer object to prettify depth output
colorizer = realsense.colorizer();

% define point cloud object
pcl_obj = realsense.pointcloud();

% Start streaming with default settings
profile = pipe.start(cfg);
alignto = realsense.stream.color;
alignedFs = realsense.align(alignto);

% Get streaming device's name
%dev = profile.get_device();
%name = dev.get_info(realsense.camera_info.name);

% Get frames. We discard the first couple to allow
% the camera time to settle

for i = 1:5
fs = pipe.wait_for_frames();
end

%create a point cloud player
%player1 = pcplayer([-1 1],[-1 1],[-1 1]);
x_error=0;
y_error=0;
frameCount =0;

%%
while(frameCount<=2000)
tic
%try
frameCount = frameCount+1;
fs = pipe.wait_for_frames();

%align the depth frames to the color stream
aligned_frames = alignedFs.process(fs);
depth = aligned_frames.get_depth_frame();
color = fs.get_color_frame();

%get the points cloud based on the aligned depth stream
if (depth.logical())
    pnts = pcl_obj.calculate(depth);   %%%% this is where I get my error************************************************
    
    if (pnts.logical())
        pcl_obj.map_to(color);
        colordata = color.get_data();
        colordatavector = [colordata(1:3:end)',colordata(2:3:end)',colordata(3:3:end)']; %HS [r g b]
        
        vertices = pnts.get_vertices();  % HS [x y z]
        %view(player1,vertices,colordatavector)
        % Colorize depth frame
        color_d = colorizer.colorize(depth);
        
        % Get actual data and convert into a format imshow can use
        % (Color data arrives as [R, G, B, R, G, B, ...] vector)
        data_d = color_d.get_data();
        img = permute(reshape(data_d',[3,color.get_width(),color.get_height()]),[3 2 1]);
        
        c_color = fs.get_color_frame();
        %Get actual data and convert into a format imshow can use
        data_c = c_color.get_data(); %data_c = colordata
        %(Color data arrives as [R, G, B, R, G, B, ...] vector)
        img_c = permute(reshape(data_c',[3,c_color.get_width(),c_color.get_height()]),[3 2 1]);
        
        %Display the image
        figure(1)
        % Display colour image
        subplot(2,1,1);
        imshow(img_c)
        
        % Display depth image
        subplot(2,1,2);
        imshow(img);
    end
end

end

error msg

Error in realsense.pointcloud/calculate (line 29)
out = realsense.librealsense_mex('rs2::pointcloud', 'calculate', this.objectHandle,
depth.objectHandle);

[MartyG-RealSense]

Yes, I realised that it did not have alignment and deleted the comment.

I researched the subject again carefully but could not find information that would be helpful. I will therefore seek assistance within Intel. I apologise for the delay that this will cause to your work in the meantime.

[hshorty]

@MartyG-RealSense Thanks (y)

[MartyG-RealSense]

Something that you could also do is try re-calibrating your D435 to eliminate the possibility that the apparent alignment problems are not due to camera mis-calibration.

[RealSenseSupport]

Hi @hshorty , I have run your script with the latest librealsense (development branch). I can't reproduce the issue.
I guess it's an issue about USB connection, or something else. Would you have a try with the latest librealsense?

[RealSenseSupport]

@hshorty Did you get chance to try the latest librealsense v2.38.1? Looking forward to your update. Thanks!

[hshorty]

Hi, Thanks for the updat. I will try the new update and let you know. I am currently using an error handling loop to stop getting crashed, but definitely I will try the new update. Regards, hs Get Outlook for Android<https://aka.ms/ghei36>

…

________________________________ From: RealSenseSupport <[email protected]> Sent: Friday, September 4, 2020, 11:13 PM To: IntelRealSense/librealsense Cc: hshorty; Mention Subject: Re: [IntelRealSense/librealsense] Getting the 'vertices' of my D435 Depth Camera after aligning the depth frame to colour frame MATLAB (#6763) @hshorty<https://github.com/hshorty> Did you get chance to try the latest librealsense v2.38.1? Looking forward to your update. Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#6763 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AP5YOXTX4S6ZSDE3RTIW5WTSEDRW3ANCNFSM4ORLNRBA>.

[RealSenseSupport]

@hshorty Any update from your side? Thanks!

[RealSenseSupport]

@hshorty Any chance to try the new update? Looking forward to your reply. Thanks!

[RealSenseSupport]

@hshorty Did you get it through? Thanks!

[RealSenseSupport]

@hshorty Sorry that we didn't get update from you for weeks. Will close it at this point. Please feel free to create new one if you still have questions or issues. Thanks!