D455 changing resolution and fps then depth stream start failure

[chenjunnn]

ROS Version: Galactic

Reproduce: ros2 launch realsense2_camera rs_launch.py align_depth:=true color_width:=640 color_height:=360 color_fps:=90.0 depth_width:=848 depth_height:=480 depth_fps:=90.0 initial_reset:=true

When I tried a lot of times, it might work once. But usually, it would be:

Firstly, lots of
[realsense2_camera_node-1] 07/09 17:38:46,957 ERROR [140509972518656] (uvc-streamer.cpp:106) uvc streamer watchdog triggered on endpoint: 130
[realsense2_camera_node-1] 07/09 17:38:46,995 WARNING [140510106736384] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[realsense2_camera_node-1] 07/09 17:38:47,291 ERROR [140509972518656] (uvc-streamer.cpp:106) uvc streamer watchdog triggered on endpoint: 130

Then
[realsense2_camera_node-1] [WARN] [1631007527.596733736] [RealSenseCameraNode]: Hardware Notification:Right MIPI error,1.63101e+12,Error,Hardware Error
[realsense2_camera_node-1] [WARN] [1631007528.598045021] [RealSenseCameraNode]: Hardware Notification:Depth stream start failure,1.63101e+12,Error,Hardware Error
[realsense2_camera_node-1] [WARN] [1631007530.600105719] [RealSenseCameraNode]: Hardware Notification:Depth stream start failure,1.63101e+12,Error,Hardware Error

And I can't see the depth stream in rviz2 or rqt_image_view.

By the way, the camera D455 work well in this setting color_width:=640 color_height:=360 color_fps:=90.0 depth_width:=848 depth_height:=480 depth_fps:=90.0 in realsense-viewer.

[MartyG-RealSense]

Hi @sink0918 Which RealSense ROS2 wrapper version and librealsense SDK version are you using, please? Support for ROS2 Galactic was first introduced in the most recent ROS wrapper version at the time of writing this, which is 3.2.2. This particular ROS wrapper version should be matched with librealsense SDK version 2.48.0 (there is not currently a wrapper version designed for librealsense 2.49.0).

https://github.com/IntelRealSense/realsense-ros/releases/tag/3.2.2

[chenjunnn]

[MartyG-RealSense]

Thanks very much for the log image. Your wrapper and librealsense SDK versions are correctly matched, the camera is being correctly identified as being on a USB 3.2 connection, and your custom stream configurations for depth and color have been accepted by the launch.

Below the bottom line of the image - RealSense Node is up! - are there warning messages such as control_transfer returned continuously generating, or are there just a few messages and then they stop appearing?

[chenjunnn]

It keeps continuously generating, and when I use rqt_image_view or rviz2 to see the depth image, the Depth stream start failure message would appear as I mentioned before #2068 (comment)

[MartyG-RealSense]

If the control_transfer returned warning is continuously generating like this then it can indicate that there is a serious communication problem with the device. In this particular case that problem seems to be limited to the RealSense ROS wrapper though, as the camera was performing normally for you in the RealSense Viewer program.

Could you try changing the color and depth FPS in the launch from 90 FPS to 30 please to test whether the warning stops generating rapidly?

[chenjunnn]

It works now. But I still need 90fps image in my program, is there any solution?
(Plus: 60 fps crushed too)

[MartyG-RealSense]

Does it help if you set RViz to use compressed data instead of raw, as described in #1510 (comment)

[chenjunnn]

rviz2 does not have this option.

And I want to mention that sometimes this resolution and fps can work. I think there may be something wrong about the ROS2 Wrapper.

Also the color stream works totally fine with 90fps in 640x360, and the one fails always would be the depth stream(even if I decrease depth solution)

[MartyG-RealSense]

Thanks very much for the images. Does reliability of depth streaming improve or worsen if you remove initial_reset:=true from your launch instruction?

[chenjunnn]

I tested 10 times for each situation, here is the result

• without initial_reset:=true depth stream crushed 9 times.
  And the only time the depth stream published in 90hz was very strange.
  

• with initial_reset:=true depth stream crushed 7 times, there were 3 times it worked well.
  

So the conclusion is initial_reset:=true would help improve depth streaming.

[MartyG-RealSense]

Doronhi the RealSense ROS wrapper developer suggests in #1040 that if you are able to stream depth and color at the rate expected, such as 90, then problems with doing so may be related to the capabilities of the computer hardware being used with the camera. What computer / computing device are you using, please?

[chenjunnn]

intel NUC8 i5-8259U

[MartyG-RealSense]

That is an 8th generation i5 processor, so it should be a capable PC in terms of processor power.

The majority of RealSense ROS users do not set their FPS above 30, so there are few references about the performance that can be expected with 60 or 90.

In the rs_launch.py launch file, it is indicated that the infrared channels infra1 and infra2 are enabled.

https://github.com/IntelRealSense/realsense-ros/blob/ros2/realsense2_camera/launch/rs_launch.py#L42-L43

If you do not require infrared then you could therefore try disabling the infra streams to see whether it improves performance by adding the commands below to your roslaunch instruction:

enable_infra1:=false enable_infra2:=false

For example:

ros2 launch realsense2_camera rs_launch.py align_depth:=true color_width:=640 color_height:=360 color_fps:=90.0 depth_width:=848 depth_height:=480 depth_fps:=90.0 initial_reset:=true enable_infra1:=false enable_infra2:=false

[chenjunnn]

I have tried
ros2 launch realsense2_camera rs_launch.py align_depth:=true color_width:=640 color_height:=360 color_fps:=90.0 depth_width:=848 depth_height:=480 depth_fps:=90.0 initial_reset:=true enable_infra1:=false enable_infra2:=false
but the problem still exists.

And I wonder if it is the performance problem, why there will be chances that 90 FPS streams work perfectly well?

[MartyG-RealSense]

Could you monitor what the CPU usage percentage is when launching to see how much load is being placed on the CPU. In Windows this can be done in the Windows Task Manager interface, under the 'Performance' tab. If you are using Ubuntu then a system monitoring tool such as htop could be used.

[chenjunnn]

I think the CPU usage percentage is within the normal range.

• using 90fps (depth stream crushed) :
  

• using 60fps (it worked well) :
  

[MartyG-RealSense]

I conducted extensive further research but do not have many other suggestions to make. Because 90 FPS is used so few times in the history of the RealSense ROS wrapper, I don't feel that I can give guarantees that this speed will be achievable with stability in the wrapper when using it with both depth & color and with alignment.

Do you see any improvement at 90 FPS if you add the term enable_sync:=true to your roslaunch instruction? This command gathers closest frames of different sensors, infrared, color and depth, to be sent with the same timetag.

[chenjunnn]

My launching command is

ros2 launch realsense2_camera rs_launch.py align_depth:=true color_width:=640 color_height:=360 color_fps:=90.0 depth_width:=640 depth_height:=480 depth_fps:=90.0 enable_sync:=true enable_gyro:=true enable_accel:=true unite_imu_method:='linear_interpolation' enable_infra1:=false enable_infra2:=false

and there is no improvement.

[MartyG-RealSense]

At this point I will highlight your case to Doronhi the RealSense ROS wrapper developer to seek input. Thanks very much for your patience!

[MartyG-RealSense]

Hi @sink0918 I have received advice from Doronhi. May I ask which installation methods you used to install the librealsense SDK and the RealSense ROS wrapper, please?

[chenjunnn]

Firstly I install RealSense ROS wrapper using
sudo apt-get install ros-foxy-realsense2-camera

Then install librealsense using (in order to use realsense-viewer)
sudo apt-get install librealsense2-dkms
sudo apt-get install librealsense2-utils

[MartyG-RealSense]

This sounds like what Doronhi thought you may have done in the advice that he provided (install the ROS wrapper from Debian packages and separately install librealsense from packages).

The correct order of installation when installing librealsense and the ROS wrapper separately is to install librealsense first and the ROS wrapper secondly. So if you installed librealsense again after having installed the SDK and wrapper together with Method 1, then the ROS wrapper would need to be built again from source code using Method 2. This is because each time that the librealsense version is changed, the ROS wrapper has to be built again.

[chenjunnn]

I have tried the Method 2, building it from the source code, (also tested purging librealsense and ros2 wrapper then re-installing them in the correct order you mentioned using apt) but there is no improvement.

Could you please test it on your device to see if it is possible to use D455 in ROS2 wrapper with both color and depth streams of 90fps?

[MartyG-RealSense]

I will pass your results back to Doronhi to seek further advice. Thanks very much for your patience.

[doronhi]

Hi guys. As I don't currently have a NUC at my disposal I tested it using a VMware machine installed on my laptop (Intel i7-8850H with 48GB RAM).
I seem to get the results as you - While using the RSUSB backend (as in the ros installation) I didn't even get 30 fps. But, using v4l backend I got about 90 fps:

doronhi@ubuntu:~/ros2_ws$ ros2 topic hz /camera/aligned_depth_to_color/image_raw
average rate: 85.990
	min: 0.010s max: 0.033s std dev: 0.00333s window: 87
average rate: 85.936
	min: 0.010s max: 0.034s std dev: 0.00335s window: 173

@sink0918 , Do you see the same error messages when you run realsense2_camera as you did at the beginning of this post, before you reinstalled the wrapper?
Could you please post the output of the following commands: apt list --installed | grep realsense, which realsense-viewer?
I am just trying to make sure where we stand.
Thanks

[chenjunnn]

And I don't understand what error messages you referred it is.

[doronhi]

You should remove the debian installation of both realsense2-camera and ros-XXX-librealsense:

sudo apt purge ros-foxy-realsense2-camera
sudo apt purge ros-foxy-realsense2-camera-msgs
sudo apt purge ros-foxy-librealsense2
sudo apt purge ros-galactic-realsense2-camera
sudo apt purge ros-galactic-realsense2-camera-msgs
sudo apt purge ros-galactic-librealsense2

Then rebuild from source - delete your "build" and "install" directories in the ros2_ws directory and build - colcon build.
It is supposed to fail. If it fails it's a good sign - it means you don't have another librealsense2 version installed (Like you would have if you also built librealsense2 from code).
Now - Install the librealsense2-dev package:
sudo apt install librealsense2-dev
And then try again to build the realsense2_camera - colcon build

[chenjunnn]

It works! Thank you! @doronhi

But I have one more question, now the version of LibRealSense that RealSenseCameraNode runs with is v2.49.0. So if I want to install the ros2 wrapper on a brand new device to use realsense with 90 FPS streams, does it mean that I have to build it from source code with the latest LibRealSense?

Also, I want to make sure that what caused my problem.

Thanks again.

[doronhi]

As for making sure that that's what caused the problem - it fits well with my guess and reproduction so we can assume the root cause is using RSUSB backend instead of v4l backend.
The ros-librealsense2 package is built with RSUSB backend so you should avoid using it. That means that you have to use the librealsense2-dev package which is built with v4l backend.
Unfortunately for you, it means that on new devices you will have to build realsense2_camera from source code.
The version of librealsense2 - 2.49.0 or 2.48.0 is less important in that context.
I'm glad it works and sorry for the inconvenience.

[chenjunnn]

Thank you for your answer.
And I also want to know that
Will this problem be fixed in the future version of ros-librealsense2 package? @doronhi

[doronhi]

Hi,
In the phrase "this problem" I can see 2 possibilities:

1. Will the RSUSB backend be improved to allow the same performance as the native v4l backend?
2. Will future ros-librealsense2 packages will be built with v4l backend, as the regular librealsense2 Debian packages do?

Unfortunately, at this point, I don't know.
There is currently no work being done to improve the performance of the RSUSB backend.
There is no definitive answer to the question of which backend is better to be used in the Debian package and I am not aware of any ongoing discussion about it right now.

[chenjunnn]

Here is a new issue with the v4l backend, when I check the imu data (using unite_imu_method: linear_interpolation), it can't be transmitted smoothly.

Example (at about 9s in the clip) :

camera.yaml.-.ros2_ws.SSH_.192.168.1.11.-.Visual.Studio.Code.2021-09-23.23-47-50.mp4

[MartyG-RealSense]

Hi @sink0918 Does your IMU performance improve if you set unite_imu_method to copy instead of linear_interpolation

[chenjunnn]

Hi @sink0918 Does your IMU performance improve if you set unite_imu_method to copy instead of linear_interpolation

Yes, using copy dose improve.

[MartyG-RealSense]

Excellent news that using 'copy' was helpful, @sink0918 :)

[MartyG-RealSense]

Hi @sink0918 Do you require further assistance with this case, please? Thanks!

[chenjunnn]

Hi @sink0918 Do you require further assistance with this case, please? Thanks!

I'd like to know why unite_imu_method: linear_interpolation can't work in this situation(v4l backend). Thank you for your help.

[MartyG-RealSense]

My understanding is that when using linear_interpolation, the message is liable to vary, whereas copy is more stable. Doronhi provides further description of the differences between the two modes in #598 (comment)

[MartyG-RealSense]

Hi @sink0918 Do you require further assistance with this case, please? Thanks!

[chenjunnn]

I have no further question, thank you for your support! @MartyG-RealSense

[MartyG-RealSense]

Thanks very much @sink0918 for the update! As you have no further questions, I will close the case. Thanks again :)