Empty point cloud on rs2::pointcloud::calculate()

[AaronZettler]

Required Info
Camera Model	D435i
Firmware Version	05.12.01.00
Operating System & Version	Ubuntu 18.04.3 LTS
Kernel Version (Linux Only)	5.3.0-26-generic
Platform	PC (VMware)
SDK Version	2.31.0
Language	C++
Segment	Robot

I was referred to this issue board by MartyG at https://support.intelrealsense.com/.

Issue Description

I am trying to get the vertices of a point-cloud but no matter what I do, I always get [x=0, y=0, z=0] for all points. I am using a .bag file hat was recorded with a realsense d435i camera.

My Test-Application:

Application:

#include <librealsense2/rs.hpp>
#include <algorithm>
#include <iostream>

int main(int argc, char * argv[]) {
  rs2::pointcloud pc;
  rs2::points points;
  rs2::pipeline pipe;

  rs2::config cfg;
  cfg.enable_device_from_file("recording.bag");
  pipe.start(cfg);

  int frame_off_interest = 100;
  while (1) {
    if (frame_off_interest <= 0) {
      auto frames = pipe.wait_for_frames();
      auto color = frames.get_color_frame();
      auto depth = frames.get_depth_frame();

      std::cout << "Measurement at [300, 300] = " << std::to_string(depth.get_distance(300, 300)) << std::endl;

      pc.map_to(color);
      points = pc.calculate(depth);
      auto vertices = points.get_vertices();

      std::cout << "Fond " << std::to_string(points.size()) << " Vertices" << std::endl;

      int vertex_count = 0;
      for (int i = 0; i < points.size(); i++) {
        if (vertices[i].x > 0.0 || vertices[i].y > 0.0 || vertices[i].z > 0.0) {
          vertex_count++;
        }
      }
      std::cout << "Count Vertices that are not (0, 0, 0) = " << std::to_string(vertex_count) << std::endl;
      break;
    }
    frame_off_interest--;
  }
  return EXIT_SUCCESS;
}

CMakeLists.txt:

cmake_minimum_required(VERSION 2.8 FATAL_ERROR)
project(LRS_PCL)

#used for intelisense by vscode
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)

find_package(realsense2 REQUIRED)
include_directories(${realsense2_INCLUDE_DIRS})
set(CMAKE_CXX_STANDARD 11)
set(CMAKE_CXX_STANDARD_REQUIRED TRUE)

add_executable (LRS_PCL rs-pointcloud.cpp)
target_link_libraries (LRS_PCL ${realsense2_LIBRARY})

When I execute the test application, the depth-map has data, but I always get [x=0, y=0, z=0] for all points. See the output of my Test-Application below:

Measurement at [300, 300] = 2.634000
Fond 921600 Vertices
Count Vertices that are not (0, 0, 0) = 0

I would be grateful for your help.

[RealSenseCustomerSupport]

---

@AaronZettler Could you please try RealSenseViewer to see if you can get normal 3D preview? Also noticed that you're using SDK 2.9.1 which is old version. Please upgrade to the latest SDK version try again.

[RealSenseCustomerSupport]

---

@AaronZettle Any update from your side? Looking forward to your reply. Thanks!

[RealSenseCustomerSupport]

---

@AaronZettle Could you please update? Thanks!

[AaronZettler]

Sorry for taking so long to respond. I am actually using SDK version 2.31.0. My goal is to develop and test an Application on my VM that will later run on the JetsonNano Devkit. The strange thing is that this problem only accrues when i run the sample-code above on the VM. On the JetsonNano everything works like it is supposed to.

I used the buildLibrealsense.sh script from the JetsonHacksNano repository to build and install the SDK on both environments.

My temporary solution for this problem is that, if i compile my application on the VM, i do the conversion manually.

std::vector<std::array<float, 3>> points;
std::vector<std::array<uint8_t, 3>> colors;

cv::Mat cop_col = colimg.clone();

for (int j = 0; j < depth->get_width(); j++)
{
    for (int i = 0; i < depth->get_height(); i++)
    {
        float point[3];
        float pixel[2] = {(float)j, (float)i};

        rs2_intrinsics intr = depth->get_profile().as<rs2::video_stream_profile>().get_intrinsics();
        rs2_deproject_pixel_to_point(point, &intr, pixel, depth->get_distance(pixel[0], pixel[1]));

        std::array<float, 3> point_data;
        std::array<uint8_t, 3> color_data;

        point_data = {point[0], point[1], point[2]};
        points.push_back(point_data);

        cv::Vec3b color = cop_col.at<cv::Vec3b>(i, j);
        color_data = {color[0], color[1], color[2]}; //{col[3*(i * stride + j)], col[3*(i * stride + j) + 1], col[3*(i * stride + j) + 2]};
        colors.push_back(color_data);
    }
}

pcl::PointCloud<pcl::PointXYZRGB>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZRGB>);

cloud->width = static_cast<uint32_t>(widht);
cloud->height = static_cast<uint32_t>(height);
cloud->is_dense = false;
cloud->points.resize(widht * height);

for (int i = 0; i < widht * height; i++)
{
    cloud->points[i].x = points[i][0];
    cloud->points[i].y = points[i][1];
    cloud->points[i].z = points[i][2];

    cloud->points[i].r = colors[i][0];
    cloud->points[i].g = colors[i][1];
    cloud->points[i].b = colors[i][2];
}

Because doing it this way is much slower and the color image is not aligned correctly, i'm still looking for a solution for this problem.

[RealSenseCustomerSupport]

---

@AaronZettler As said in https://github.com/IntelRealSense/librealsense/wiki/Troubleshooting-Q&A#q-im-using-a-virtual-machine-and-the-camera-is-not-working due to the USB 3.0 translation layer between native hardware and virtual machine, the librealsense team does not support installation in a VM. Could you please try without VM? Thanks!

[RealSenseCustomerSupport]

---

@AaronZettler Any update from your side? Thanks!

[RealSenseCustomerSupport]

---

@AaronZettler Could you please update? Thanks!

[RealSenseCustomerSupport]

---

@AaronZettler Sorry that we didn't get response from you for weeks. Will close it if no other questions. Please feel free to create another new one if you still have issues or questions. Thanks!