problems while using zenoh with simulation

[padhupradheep]

I was trying to launch the gz simulation using the command:

ros2 launch rox_bringup bringup_sim_launch.py [jazzy-devel branch]

if you want to use this, then you also need to export the gz resource path like this : export GZ_SIM_RESOURCE_PATH=~/rolling_ws/src/rox/:~/rolling_ws/src/neo_gz_worlds/models

and also clone the neo_gz_worlds

while doing so, the spawn create node fails for some reason. with the following error:

[ERROR] [create-2]: process has died [pid 48936, exit code -6, cmd '/opt/ros/rolling/lib/ros_gz_sim/create -topic robot_description -name rox --ros-args -r __node:=spawn_model'].

Also on the teleop terminal, I can see the following piece of warning:

[WARN] [1719424825.516089397] [rmw_zenoh_cpp]: Received liveliness token to remove node /spawn_model from the graph before all pub/subs/clients/services for this node have been removed. Removing all entities first...
 [WARN] [1719424825.516213720] [rmw_zenoh_cpp]: Received liveliness token to remove unknown node /spawn_model from the graph. Ignoring...
 [WARN] [1719424825.516240348] [rmw_zenoh_cpp]: Received liveliness token to remove unknown node /spawn_model from the graph. Ignoring...
 [WARN] [1719424825.516262597] [rmw_zenoh_cpp]: Received liveliness token to remove unknown node /spawn_model from the graph. Ignoring...
 [WARN] [1719424825.516287238] [rmw_zenoh_cpp]: Received liveliness token to remove unknown node /spawn_model from the graph. Ignoring...
 [WARN] [1719424825.516309925] [rmw_zenoh_cpp]: Received liveliness token to remove unknown node /spawn_model from the graph. Ignoring...
[WARN] [1719424825.516332366] [rmw_zenoh_cpp]: Received liveliness token to remove unknown node /spawn_model from the graph. Ignoring..

we don't see this while using other dds vendors.

Although we are able to move the robot like as usual, I thought, this might be something helpful to improve zenoh

Here is the full logs:

[INFO] [launch]: All log files can be found below /home/pradheep/.ros/log/2024-06-26-19-55-00-488320-pradheep-ThinkPad-P52-48928
[INFO] [launch]: Default logging verbosity is set to INFO
ign_args is deprecated, migrate to gz_args!
[INFO] [robot_state_publisher-1]: process started with pid [48935]
[INFO] [create-2]: process started with pid [48936]
[INFO] [ruby $(which gz) sim-3]: process started with pid [48937]
[INFO] [parameter_bridge-4]: process started with pid [48938]
[INFO] [xterm-5]: process started with pid [48939]
[robot_state_publisher-1] [INFO] [1719424501.240379637] [rmw_zenoh_cpp]: Successfully connected to a Zenoh router with id f6b05922ea5b0268dfc539c643e22fc.
[create-2] [INFO] [1719424501.258359164] [rmw_zenoh_cpp]: Successfully connected to a Zenoh router with id f6b05922ea5b0268dfc539c643e22fc.
[parameter_bridge-4] [INFO] [1719424501.275260648] [rmw_zenoh_cpp]: Successfully connected to a Zenoh router with id f6b05922ea5b0268dfc539c643e22fc.
[robot_state_publisher-1] [INFO] [1719424502.253281976] [robot_state_publisher]: Robot initialized
[create-2] [INFO] [1719424502.264140176] [spawn_model]: Requesting list of world names.
[create-2] [INFO] [1719424502.512878454] [spawn_model]: Waiting messages on topic [robot_description].
[create-2] [INFO] [1719424502.517359768] [spawn_model]: Requested creation of entity.
[create-2] [INFO] [1719424502.517446780] [spawn_model]: OK creation of entity.
[create-2] thread '<unnamed>' panicked at /build/rustc-Px5Ywq/rustc-1.75.0+dfsg0ubuntu1/library/std/src/thread/local.rs:246:26:
[create-2] cannot access a Thread Local Storage value during or after destruction: AccessError
[create-2] note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[ruby $(which gz) sim-3] Warning [Utils.cc:132] [/sdf/model[@name="rox"]/link[@name="base_link"]/sensor[@name="lidar_1_sensor"]/output_type:<urdf-string>:L0]: XML Element[output_type], child of element[sensor], not defined in SDF. Copying[output_type] as children of [sensor].
[ruby $(which gz) sim-3] Warning [Utils.cc:132] [/sdf/model[@name="rox"]/link[@name="base_link"]/sensor[@name="lidar_1_sensor"]/gz_frame_id:<urdf-string>:L0]: XML Element[gz_frame_id], child of element[sensor], not defined in SDF. Copying[gz_frame_id] as children of [sensor].
[ruby $(which gz) sim-3] Warning [Utils.cc:132] [/sdf/model[@name="rox"]/link[@name="base_link"]/sensor[@name="lidar_2_sensor"]/output_type:<urdf-string>:L0]: XML Element[output_type], child of element[sensor], not defined in SDF. Copying[output_type] as children of [sensor].
[ruby $(which gz) sim-3] Warning [Utils.cc:132] [/sdf/model[@name="rox"]/link[@name="base_link"]/sensor[@name="lidar_2_sensor"]/gz_frame_id:<urdf-string>:L0]: XML Element[gz_frame_id], child of element[sensor], not defined in SDF. Copying[gz_frame_id] as children of [sensor].
[ruby $(which gz) sim-3] libEGL warning: egl: failed to create dri2 screen
[ruby $(which gz) sim-3] libEGL warning: egl: failed to create dri2 screen
[ruby $(which gz) sim-3] libEGL warning: egl: failed to create dri2 screen
[parameter_bridge-4] [INFO] [1719424503.283348538] [parameter_bridge]: Creating GZ->ROS Bridge: [/odom (ignition.msgs.OdometryWithCovariance) -> /odom (nav_msgs/msg/Odometry)] (Lazy 0)
[parameter_bridge-4] [INFO] [1719424503.285428193] [parameter_bridge]: Creating GZ->ROS Bridge: [/tf (ignition.msgs.Pose_V) -> /tf (tf2_msgs/msg/TFMessage)] (Lazy 0)
[parameter_bridge-4] [INFO] [1719424503.287045061] [parameter_bridge]: Creating GZ->ROS Bridge: [/scan (ignition.msgs.LaserScan) -> lidar_1/scan_filtered (sensor_msgs/msg/LaserScan)] (Lazy 0)
[parameter_bridge-4] [INFO] [1719424503.288481351] [parameter_bridge]: Creating GZ->ROS Bridge: [/scan2 (ignition.msgs.LaserScan) -> lidar_2/scan_filtered (sensor_msgs/msg/LaserScan)] (Lazy 0)
[parameter_bridge-4] [INFO] [1719424503.289275710] [parameter_bridge]: Creating GZ->ROS Bridge: [/cmd_vel (ignition.msgs.Twist) -> /cmd_vel (geometry_msgs/msg/Twist)] (Lazy 0)
[parameter_bridge-4] [INFO] [1719424503.289839045] [parameter_bridge]: Creating ROS->GZ Bridge: [/cmd_vel (geometry_msgs/msg/Twist) -> /cmd_vel (ignition.msgs.Twist)] (Lazy 0)
[parameter_bridge-4] [INFO] [1719424503.290659195] [parameter_bridge]: Creating GZ->ROS Bridge: [clock (gz.msgs.Clock) -> /clock (rosgraph_msgs/msg/Clock)] (Lazy 0)
[ERROR] [create-2]: process has died [pid 48936, exit code -6, cmd '/opt/ros/rolling/lib/ros_gz_sim/create -topic robot_description -name rox --ros-args -r __node:=spawn_model'].
[robot_state_publisher-1] [WARN] [1719424503.638516012] [rmw_zenoh_cpp]: Received liveliness token to remove node /spawn_model from the graph before all pub/subs/clients/services for this node have been removed. Removing all entities first...
[parameter_bridge-4] [WARN] [1719424503.638518704] [rmw_zenoh_cpp]: Received liveliness token to remove node /spawn_model from the graph before all pub/subs/clients/services for this node have been removed. Removing all entities first...
[robot_state_publisher-1] [WARN] [1719424503.638619560] [rmw_zenoh_cpp]: Received liveliness token to remove unknown node /spawn_model from the graph. Ignoring...
[robot_state_publisher-1] [WARN] [1719424503.638642166] [rmw_zenoh_cpp]: Received liveliness token to remove unknown node /spawn_model from the graph. Ignoring...
[robot_state_publisher-1] [WARN] [1719424503.638661371] [rmw_zenoh_cpp]: Received liveliness token to remove unknown node /spawn_model from the graph. Ignoring...
[robot_state_publisher-1] [WARN] [1719424503.638688193] [rmw_zenoh_cpp]: Received liveliness token to remove unknown node /spawn_model from the graph. Ignoring...
[parameter_bridge-4] [WARN] [1719424503.638630572] [rmw_zenoh_cpp]: Received liveliness token to remove unknown node /spawn_model from the graph. Ignoring...
[parameter_bridge-4] [WARN] [1719424503.638659465] [rmw_zenoh_cpp]: Received liveliness token to remove unknown node /spawn_model from the graph. Ignoring...
[parameter_bridge-4] [WARN] [1719424503.638690012] [rmw_zenoh_cpp]: Received liveliness token to remove unknown node /spawn_model from the graph. Ignoring...
[parameter_bridge-4] [WARN] [1719424503.638716938] [rmw_zenoh_cpp]: Received liveliness token to remove unknown node /spawn_model from the graph. Ignoring...
[ruby $(which gz) sim-3] libEGL warning: egl: failed to create dri2 screen
[ruby $(which gz) sim-3] libEGL warning: egl: failed to create dri2 screen
[ruby $(which gz) sim-3] libEGL warning: egl: failed to create dri2 screen
[ruby $(which gz) sim-3] Warning [Utils.cc:132] [/sdf/model[@name="rox"]/link[@name="base_link"]/sensor[@name="lidar_1_sensor"]/output_type:<data-string>:L279]: XML Element[output_type], child of element[sensor], not defined in SDF. Copying[output_type] as children of [sensor].
[ruby $(which gz) sim-3] Warning [Utils.cc:132] [/sdf/model[@name="rox"]/link[@name="base_link"]/sensor[@name="lidar_1_sensor"]/gz_frame_id:<data-string>:L280]: XML Element[gz_frame_id], child of element[sensor], not defined in SDF. Copying[gz_frame_id] as children of [sensor].
[ruby $(which gz) sim-3] Warning [Utils.cc:132] [/sdf/model[@name="rox"]/link[@name="base_link"]/sensor[@name="lidar_2_sensor"]/output_type:<data-string>:L308]: XML Element[output_type], child of element[sensor], not defined in SDF. Copying[output_type] as children of [sensor].
[ruby $(which gz) sim-3] Warning [Utils.cc:132] [/sdf/model[@name="rox"]/link[@name="base_link"]/sensor[@name="lidar_2_sensor"]/gz_frame_id:<data-string>:L309]: XML Element[gz_frame_id], child of element[sensor], not defined in SDF. Copying[gz_frame_id] as children of [sensor].
[ruby $(which gz) sim-3] Warning [Utils.cc:132] [/sdf/model[@name="rox"]/link[@name="base_link"]/sensor[@name="lidar_1_sensor"]/output_type:<data-string>:L279]: XML Element[output_type], child of element[sensor], not defined in SDF. Copying[output_type] as children of [sensor].
[ruby $(which gz) sim-3] Warning [Utils.cc:132] [/sdf/model[@name="rox"]/link[@name="base_link"]/sensor[@name="lidar_1_sensor"]/gz_frame_id:<data-string>:L280]: XML Element[gz_frame_id], child of element[sensor], not defined in SDF. Copying[gz_frame_id] as children of [sensor].
[ruby $(which gz) sim-3] Warning [Utils.cc:132] [/sdf/model[@name="rox"]/link[@name="base_link"]/sensor[@name="lidar_2_sensor"]/output_type:<data-string>:L308]: XML Element[output_type], child of element[sensor], not defined in SDF. Copying[output_type] as children of [sensor].
[ruby $(which gz) sim-3] Warning [Utils.cc:132] [/sdf/model[@name="rox"]/link[@name="base_link"]/sensor[@name="lidar_2_sensor"]/gz_frame_id:<data-string>:L309]: XML Element[gz_frame_id], child of element[sensor], not defined in SDF. Copying[gz_frame_id] as children of [sensor].

[JEnoch]

The warnings about liveliness tokens are a direct consequence of the create-2 process (i.e. spawn_model node) crash:

[create-2] [INFO] [1719424502.517446780] [spawn_model]: OK creation of entity.
[create-2] thread '<unnamed>' panicked at /build/rustc-Px5Ywq/rustc-1.75.0+dfsg0ubuntu1/library/std/src/thread/local.rs:246:26:
[create-2] cannot access a Thread Local Storage value during or after destruction: AccessError
[create-2] note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[ERROR] [create-2]: process has died [pid 48936, exit code -6, cmd '/opt/ros/rolling/lib/ros_gz_sim/create -topic robot_description -name rox --ros-args -r __node:=spawn_model'].

The node didn't perform a clean exit, removing all its pub/subs/clients/services before to stop.
The Zenoh liveliness tokens are used for ROS graph discovery.

Please, reproduce the issue with the RUST_BACKTRACE=1 environment variable set for this process (or for the full host if simpler) and copy here the backtrace you get .

[padhupradheep]

Sure! Here is the back trace for you..

[create-2] thread '<unnamed>' panicked at /build/rustc-Px5Ywq/rustc-1.75.0+dfsg0ubuntu1/library/std/src/thread/local.rs:246:26:
[create-2] cannot access a Thread Local Storage value during or after destruction: AccessError
[create-2] stack backtrace:
[create-2]    0: rust_begin_unwind
[create-2]              at /build/rustc-Px5Ywq/rustc-1.75.0+dfsg0ubuntu1/library/std/src/panicking.rs:645:5
[create-2]    1: core::panicking::panic_fmt
[create-2]              at /build/rustc-Px5Ywq/rustc-1.75.0+dfsg0ubuntu1/library/core/src/panicking.rs:72:14
[create-2]    2: core::result::unwrap_failed
[create-2]              at /build/rustc-Px5Ywq/rustc-1.75.0+dfsg0ubuntu1/library/core/src/result.rs:1653:5
[create-2]    3: z_close
[create-2]    4: rmw_shutdown
[create-2]    5: rcl_shutdown
[create-2]    6: _ZN6rclcpp7Context8shutdownERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
[create-2]    7: _ZN6rclcpp7ContextD1Ev
[create-2]    8: <unknown>
[create-2]    9: <unknown>
[create-2]   10: exit
[create-2]   11: <unknown>
[create-2]   12: __libc_start_main
[create-2]   13: _start

[JEnoch]

This is probably the same issue than ros2/rmw_zenoh#170 .
As you are able to move the robot like as usual, I guess this error at exit doesn't prevent the create node to complete its tasks, right ? If yes, I think you can ignore this error that hopefully will be fixed soon.

[padhupradheep]

I guess this error at exit doesn't prevent the create node to complete its tasks, right ?

That's correct.

Then alright. I will proceed with the other tests then. Closing this issue .