Isaac ROS Nvblox contains ROS 2 packages for 3D reconstruction and cost maps for navigation. nvblox_nav2 processes depth and pose to reconstruct a 3D scene in real-time and a 2D costmap for Nav2. The costmap is used in planning during navigation as a vision-based solution to avoid obstacles.
nvblox_nav2 is designed to work with stereo cameras, which provide a depth image, and the corresponding pose uses GPU acceleration to compute 3D reconstruction and 2D costmaps using nvblox.
Above is a typical graph that uses nvblox_nav2. The input color image corresponding to the depth image is processed with unet, using the PeopleSemSegNet DNN model to estimate a segmentation mask for persons in the color image. Pose corresponding to the depth image is computed using visual_slam. The resulting persons mask and pose is used with the color image and depth image to perform 3D scene reconstruction. The output costmap is provided through a cost map plugin into Nav2, with an optional colorized 3D reconstruction into a Rviz using the mesh visualization plugin.
nvblox_nav2 builds the reconstructed map in the form of a TSDF (Truncated Signed Distance Function) stored in a 3D voxel grid. This approach is similar to 3D occupancy grid mapping approaches in which occupancy probabilities are stored at each voxel. However, TSDF-based approaches like nvblox store the (signed) distance to the closest surface at each voxel. The surface of the environment can then be extracted as the zero-level set of this voxelized function. Typically, TSDF-based reconstructions provide higher quality surface reconstructions.
In addition to their use in reconstruction, distance fields are also useful for path planning because they provide an immediate means of checking whether potential future robot positions are in collision.
People are common obstacles for mobile robots, and while part of a costmap, people should not be part of the 3D reconstruction. Planners that provide behavioral awareness by navigating differently depending on their proximity to people, benefit from a costmap for people. Person segmentation is computed using the color image, with the resulting mask applied to the depth image separating depth into scene depth and person depth images. The scene depth image is forwarded to TSDF mapping as explained above, the depth image for people is processed to an occupancy grid map.
To relax the assumption that occupancy grid maps only capture static objects, Nvblox applies an occupancy decay step. At a fixed frequency, all voxel occupancy probabilities are decayed towards 0.5 over time. This means that the state of the map (occupied or free) becomes less certain after it has fallen out of the field of view, until it becomes unknown (0.5 occupancy probability).
- Isaac ROS Nvblox
Update 2023-04-05: Human reconstruction and new weighting functions.
This package is designed and tested to be compatible with ROS 2 Humble running on Jetson or an x86_64 system with an NVIDIA GPU.
Note: Versions of ROS 2 earlier than Humble are not supported. This package depends on specific ROS 2 implementation features that were only introduced beginning with the Humble release.
| Platform | Hardware | Software | Notes |
|---|---|---|---|
| Jetson | Jetson Orin Jetson Xavier |
JetPack 5.1.1 | For best performance, ensure that power settings are configured appropriately. |
| x86_64 | NVIDIA GPU | Ubuntu 20.04+ CUDA 11.8+ |
To simplify development, we strongly recommend leveraging the Isaac ROS Dev Docker images by following these steps. This will streamline your development environment setup with the correct versions of dependencies on both Jetson and x86_64 platforms.
Note: All Isaac ROS Quickstarts, tutorials, and examples have been designed with the Isaac ROS Docker images as a prerequisite.
-
Set up your development environment by following the instructions here.
-
Clone this repository and its dependencies under
~/workspaces/isaac_ros-dev/srcor/ssd/workspaces/isaac_ros-dev/srcdepending upon SD card or SSD setup.cd ~/workspaces/isaac_ros-dev/src
Note: For Jetson setup with SSD as optional storage:
cd /ssd/workspaces/isaac_ros-dev/srcgit clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_common
git clone --recurse-submodules https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_nvblox && \ cd isaac_ros_nvblox && git lfs pull
-
Pull down a ROS Bag of sample data appropriately depending upon SD card or SSD setup:
cd ~/workspaces/isaac_ros-dev/src/isaac_ros_nvblox && \ git lfs pull -X "" -I "nvblox_ros/test/test_cases/rosbags/nvblox_pol"
Note: For Jetson setup with SSD as optional storage:
cd /ssd/workspaces/isaac_ros-dev/src/isaac_ros_nvblox && \ git lfs pull -X "" -I "nvblox_ros/test/test_cases/rosbags/nvblox_pol"
-
Launch the Docker container using the
run_dev.shscript (ISAAC_ROS_WSenvironment variable will take care of the correct path depending upon SD card or SSD setup as mentioned here):Note: This step requires access to the internet to be able to build and launch the Docker container properly!
cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \ ./scripts/run_dev.sh ${ISAAC_ROS_WS}
-
Inside the container, install package-specific dependencies via
rosdep:cd /workspaces/isaac_ros-dev/ && \ rosdep install -i -r --from-paths src --rosdistro humble -y --skip-keys "libopencv-dev libopencv-contrib-dev libopencv-imgproc-dev python-opencv python3-opencv nvblox"
-
Build and source the workspace:
cd /workspaces/isaac_ros-dev && \ colcon build --symlink-install && \ source install/setup.bash
-
(Optional) Run tests to verify complete and correct installation:
colcon test --executor sequential -
In the current terminal inside the Docker container, run the launch file for Nvblox with
nav2:ros2 launch nvblox_examples_bringup isaac_sim_example.launch.py
-
Open a second terminal and enter inside the running Docker container (similar to Step 4):
cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \ ./scripts/run_dev.sh ${ISAAC_ROS_WS}
-
In the second terminal inside the Docker container, play the ROS Bag:
ros2 bag play src/isaac_ros_nvblox/nvblox_ros/test/test_cases/rosbags/nvblox_pol
You should see the robot reconstructing a mesh, with the 2D ESDF slice overlaid on top.
The launch files for all examples are available in the nvblox_examples_bringup package:
| Launch file | Arguments | Description |
|---|---|---|
isaac_sim_example.launch.py |
run_nav2, run_rviz |
Example to run with Isaac Sim (tutorial) |
isaac_sim_humans_example.launch.py |
run_nav2, run_rviz |
Example to run with Isaac Sim including human reconstruction (tutorial) |
realsense_example.launch.py |
from_bag, bag_path, run_rviz |
Example to run with a realsense camera (tutorial) |
realsense_humans_example.launch.py |
from_bag, bag_path, run_rviz |
Example to run with a realsense camera including human reconstruction (tutorial) |
record_realsense.launch.py |
launch_realsense, run_rqt |
Record realsense data to replay with the above examples (tutorial) |
To customize your development environment, reference this guide.
- isaac_ros_nvblox: A meta-package.
- nvblox_examples_bringup: Launch files and configurations for launching the examples.
- nvblox_image_padding: Node to pad and crop images for adjusting the image size to the fixed input resolution that is required by the image segmentation network.
- nvblox_isaac_sim: Contains scripts for launching Isaac Sim configured for use with nvblox.
- realsense_splitter: Node for using the realsense camera with inbuilt projector. See why this is needed here.
- semantic_label_conversion: Package for converting semantic labels coming from Isaac Sim to mask images used by nvblox (readme).
- nvblox_msgs: Custom messages for transmitting the output distance map slice and mesh over ROS 2.
- nvblox_nav2: Contains a custom plugin that allows ROS 2 Nav2 to consume nvblox distance map outputs.
- nvblox_performance_measurement: Multiple packages containing tools for measuring nvblox performance (readme).
- nvblox_ros: The ROS 2 wrapper for the core reconstruction library and the nvblox node.
- nvblox_ros_common: Package providing repository wide utility functions.
- nvblox_rviz_plugin: A plugin for displaying nvblox's (custom) mesh type in RVIZ.
- [submodule] nvblox: The core (ROS independent) reconstruction library.
Find all available ROS 2 parameters here.
Find all ROS 2 subscribers, publishers and services here.
For solutions to problems with Isaac ROS, please check here.
Please follow the workaround here.
See our troubleshooting page here.
If it looks like you are dropping messages or you are not receiving any messages, please consult our troubleshooting page here.
| Date | Changes |
|---|---|
| 2023-04-05 | Human reconstruction and new weighting functions. |
| 2022-12-10 | Updated documentation. |
| 2022-10-19 | Updated OSS licensing |
| 2022-08-31 | Update to be compatible with JetPack 5.0.2. Serialization of nvblox maps to file. Support for 3D LIDAR input and performance improvements. |
| 2022-06-30 | Support for ROS 2 Humble and miscellaneous bug fixes. |
| 2022-03-21 | Initial version. |