Point cloud visualization plugin

[chapulina]

🎉 New feature

• Closes Generic point cloud visualization gz-sim#1156
• Requires Add PointCloudPacked iterators gz-msgs#210 (also needs to be merged forward)

Summary

Porting the point cloud visualization tool from osrf/lrauv.

This needs to target Garden because it requires per-point coloring of point clouds:

• Per-point colors in point clouds gz-rendering#494
• Set per-point color on markers #318
• Marker: support different colors per point gz-msgs#202

Features:

• Display hundreds of thousands of points with different colors
• Point positions in 3D come from a PointCloudPacked message
• Optionally, users can choose a separate FloatV topic to get values for each point
• Point size and color can be changed from the GUI
• Multiple point clouds can be displayed side-by-side, just instantiate the plugin multiple times

Known issues:

• [ogre2] Point SetSize API applies globally to all points gz-rendering#497

Future work:

There are many improvements that can be made to the plugin. Here are some improvements I can think of for the future:

• Use a palette with more than 2 colors to have a richer gradient
• Optionally take color from a channel on the point cloud, instead of another topic
• Support a custom reference frame, so we can properly visualize LIDAR data - right now all points must be expressed in the world frame

Test it

See the added example and its instructions:

Here's an examples using it to display maritime science data for LRAUV:

Checklist

• Signed all commits for DCO
• Added tests
• Added example and/or tutorial
• Updated documentation (as needed)
• Updated migration guide (as needed)
• codecheck passed (See contributing)
• All tests passed (See test coverage)
• While waiting for a review on your PR, please help review another open pull request to support the maintainers

Note to maintainers: Remember to use Squash-Merge and edit the commit message to match the pull request summary while retaining Signed-off-by messages.

🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸

[mabelzhang]

First pass. I looked at the code, but I need to finish rebuilding Ignition G to test this.

I know we've looked at this together a handful of times already, but I'm trying to wear a stricter hat here as it's going into Ignition, whereas in the application-specific project it's more like "anything goes."

[chapulina]

Thanks for the review, @mabelzhang . I've addressed your comments and added an example. This is now open for review.

[mabelzhang]

Thanks for adding the example! That made it a lot easier to test. Just two typos.

[mabelzhang]

Just pending green CI. Do we care about the doxygen warnings in the Bionic CI?

[chapulina]

Do we care about the doxygen warnings in the Bionic CI?

Ouch oh yeah we do, on it

[chapulina]

Just pending green CI.

Ok, doxygen should be fixed with edecdb8 . Besides that:

• Ubuntu's warnings should be fixed when Add project() to examples and remove hard-dependency on Ogre1 #345 is forward-ported
• The Helpers test failure on Homebrew looks like a new failure unrelated to this PR

[mabelzhang]

Okay re Jenkins ones.

Are these ones on the GitHub Actions intermittent?

  	 23 - UNIT_Publisher_TEST (Failed)
  	 35 - INTEGRATION_ExamplesBuild_TEST (Failed)

UNIT_Publisher_TEST is about X, failing only on the PR Action

  [GUI] [Wrn] [Application.cc:674] [QT] QXcbConnection: Could not connect to display :1.0
  [GUI] [Err] [Application.cc:678] [QT] Could not connect to any X display.

INTEGRATION_ExamplesBuild_TEST is failing on both the PR and push Actions.

[chapulina]

UNIT_Publisher_TEST is about X, failing only on the PR Action

Yup, these flaky failures due to X are all captured here:

• Deflake X-display tests on GitHub actions #58

It makes life very complicated as we have to check each test and see if the failure was due to X 😢

[chapulina]

Humm now I'm confused about the Helper test failure on Homebrew. It doesn't fail on any of our stable branches, but I can't see why this PR would affect it.

[mabelzhang]

I'm looking at the test results again, a different part of the log, and I'm seeing this related to the ExamplesBuild test that was failing:

  /github/workspace/examples/standalone/point_cloud/point_cloud.cc:31:40: error: variable 'std::atomic<bool> g_terminatePub' has initializer but incomplete type
   static std::atomic<bool> g_terminatePub(false);
                                          ^
  CMakeFiles/point_cloud.dir/build.make:62: recipe for target 'CMakeFiles/point_cloud.dir/point_cloud.cc.o' failed
  make[3]: *** [CMakeFiles/point_cloud.dir/point_cloud.cc.o] Error 1
  CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/point_cloud.dir/all' failed
  make[2]: *** [CMakeFiles/point_cloud.dir/all] Error 2
  Makefile:83: recipe for target 'all' failed
  make[1]: *** [all] Error 2

I think this is a real error?

[chapulina]

I think this is a real error?

Thanks for spotting, yes! On it

[codecov]

Codecov Report

Merging #346 (147a5e4) into main (ebbb88a) will decrease coverage by 1.92%.
The diff coverage is 17.45%.

@@            Coverage Diff             @@
##             main     #346      +/-   ##
==========================================
- Coverage   64.41%   62.49%   -1.93%     
==========================================
  Files          36       37       +1     
  Lines        4952     5164     +212     
==========================================
+ Hits         3190     3227      +37     
- Misses       1762     1937     +175     

Impacted Files	Coverage Δ
src/plugins/point_cloud/PointCloud.cc	16.82% <16.82%> (ø)
include/ignition/gui/Helpers.hh	77.77% <50.00%> (-22.23%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ebbb88a...147a5e4. Read the comment docs.

[chapulina]

I'm confused about the Helper test failure on Homebrew. It doesn't fail on any of our stable branches, but I can't see why this PR would affect it.

I pushed some bumpers to see if it prevents the test from segfaulting in a8ab1c0. It feels completely unrelated to this PR, but since the test consistently fails on this branch and consistently passes on main (search for "helper" here), I can't test if it works targeting a PR at main 🤷‍♀️

[chapulina]

I pushed some bumpers to see if it prevents the test from segfaulting in a8ab1c0.

It looks like they worked 🙏

Ok, I think we're in good shape now, phew. Thanks for being thorough!

[mabelzhang]

Thank you for making all the fixes!

(If the bumpers aren't enough, welp, a future PR will see it :D)

[scpeters]

I'm confused about the Helper test failure on Homebrew. It doesn't fail on any of our stable branches, but I can't see why this PR would affect it.

I pushed some bumpers to see if it prevents the test from segfaulting in a8ab1c0. It feels completely unrelated to this PR, but since the test consistently fails on this branch and consistently passes on main (search for "helper" here), I can't test if it works targeting a PR at main 🤷‍♀️

sadly this test is failing on main, though I don't think it was caused by this PR: #360

[dakejahl]

Support a custom reference frame, so we can properly visualize LIDAR data - right now all points must be expressed in the world frame

I am trying to visualize a point cloud from a camera sensor attached to a drone but the data is in body frame. Has this been implemented and can someone steer me to the right place to add this?

[mabelzhang]

There should be a frame ID in the header... I would think. Line 48 in examples/standalone/point_cloud/point_cloud.cc in this PR seems to suggest it's been implemented.

There's usage example in the gz-sim integration tests e.g.
https://github.com/gazebosim/gz-sim/blob/gz-sim8/test/integration/environment_preload_system.cc
https://github.com/gazebosim/gz-sim/blob/gz-sim8/test/integration/environmental_sensor_system.cc

gz-sim environment system implementation is here
https://github.com/gazebosim/gz-sim/tree/gz-sim8/src/systems/environment_preload
https://github.com/gazebosim/gz-sim/tree/gz-sim8/src/systems/environmental_sensor_system

There's a tutorial for using the environment system, in a PR gazebosim/gz-sim#1806

[dakejahl]

@mabelzhang thanks for the reply. I took a look at those files and it's not clear to me how to access reference frame information of PointCloud data within a plugin. When I search for "frame" or "model" in the src/plugins/ I don't see any other examples of converting between reference frames for sensor data attached to a moving model. The goal is to visualize obstacle detect and avoid on PX4. Thanks for the help!

Here's the model I'm using for the drone, you can see how the camera is attached
https://github.com/PX4/PX4-Autopilot/blob/main/Tools/simulation/gz/models/x500_depth/model.sdf

I have some example code where I am downsampling a camera point cloud and republishing it
dakejahl/PX4-Autopilot@02ef528

Here's a pic of a drone with a depth camera looking at a block some distance from the origin

[mabelzhang]

Hey, okay, so first, we really shouldn't be starting a new topic in a merged PR. The more correct thing to do would be to open an issue and tag this PR as the relevant source code. I replied with links to the source code because they depend on code in this PR. But with the new details you posted, it should really be in a new issue ticket for better visibility.

Please keep that in mind and open a new ticket if you post new information.

@arjo129 can people set frame ID when visualizing point clouds? That would be part of the environment system right?

[arjo129]

To be fair this has nothing to do with the environment system. We should open a ticket, I suspect point clouds don't follow frame_ids set 😱 .

[dakejahl]

Sorry about that! I figured posting here was the best way to get the attention of the person who knows! 😁

Ticket here
#595