D455 cannot be found using native backend

[yzkzero]

---

Required Info
Camera Model	D455
Firmware Version	shipped by 2.50 SDK
Operating System & Version	Linux
Kernel Version (Linux Only)	kernel 5.4.70
Platform	buildroot for iMX8
SDK Version	2.50
Language	C++
Segment	{Robot/Smartphone/VR/AR/others }

Hi,

I built librealsense.so with-DFORCE_RSUSB_BACKEND=false option. And I found D415 and D435 can be recognized by this library. But D455 cannot be found in my C++ program. Also rs-enumerate has the same result.

But if I build librealsense.so with -DFORCE_RSUSB_BACKEND=true, then all my cameras (D415, D435 & D455) can be found.

We want to use V4L backend (-DFORCE_RSUSB_BACKEND=false) because RSUSB backend is not very stable in our stress test. There are always some segmentation fault from libusb after thousands of quick camera start and stop switching. This issue also happens in JetsonNano board. So I think the issue is caused by RSUSB backend, not the OS or hardware. The V4L backend works pretty stable, but the only issue is that it does not recognize D455 camera.

Can you please help to check why V4L backend does not work for D455?
Thank you!

BTW, when any D400 camera is plugged, I can list camera devices in /dev/video[1-5]. So these cameras are all recognized by the OS and hardware.

[MartyG-RealSense]

Hi @yzkzero You could try building librealsense 2.49.0 (or 2.48 if you need to use the RealSense ROS wrapper too) to see if it provides improved RSUSB stability. 2.50.0 introduced changes to RSUSB that a couple of RealSense users reported instability with compared to no problems with RSUSB on 2.49.0.

[yzkzero]

Hi @yzkzero You could try building librealsense 2.49.0 (or 2.48 if you need to use the RealSense ROS wrapper too) to see if it provides improved RSUSB stability. 2.50.0 introduced changes to RSUSB that a couple of RealSense users reported instability with compared to no problems with RSUSB on 2.49.0.

Hi MartyG,

Thanks for the reply.
I have tried compiling 2.49 library and also rebuilt my application with SDK 2.49. The behavior is the same. D415/435 can be recognized and D455 still cannot be found.

[MartyG-RealSense]

There was another case at #9880 where a D455 was not being detected with v4L on Jetson Nano but was detected on an RSUSB build. A reason for this was not found in that particular case either.

A RealSense team member gives advice in #6837 about how to check if a patch has been applied correctly.

My understanding is that one of the advantages of V4L is that it can poll for events frequently to check for changes, whilst RSUSB does not have that mechanism and so polls the device every 5 seconds by default. This means that when setting options, a change request may sometimes fail and have to be re-tried, as discussed in #6952

Another RealSense team member advises in #6921 (comment) that a file in the SDK's source code - librealsense/src/types.h - can be edited to reduce the 5 second time period between checks on an RSUSB build, at the cost of slightly higher CPU percentage usage.

[yzkzero]

There was another case at #9880 where a D455 was not being detected with v4L on Jetson Nano but was detected on an RSUSB build. A reason for this was not found in that particular case either.

A RealSense team member gives advice in #6837 about how to check if a patch has been applied correctly.

My understanding is that one of the advantages of V4L is that it can poll for events frequently to check for changes, whilst RSUSB does not have that mechanism and so polls the device every 5 seconds by default. This means that when setting options, a change request may sometimes fail and have to be re-tried, as dsicussed in #6952

Another RealSense team member advises in #6921 (comment) that a file in the SDK's source code - librealsense/src/types.h - can be edited to reduce the 5 second time period between checks on an RSUSB build ,at the cost of slightly higher CPU percentage usage.

Hi MartyG,

In my case, the D455 camera cannot be found in my ARM64 board with buildroot linux. But D415 and D435 have no such issue.
Why I don't want to use RSUSB backend is because RSUSB is not very stable, it occasionally crashes and the error is segmentation fault in libusb (crash not only on my ARM64 board, but also on Jetson Nano board). But the same application code runs very stable if the lib is complied with native V4L backend. The only issue is that D455 cannot be found.

I'm wondering if there's anyone successfully use D455 using librealsense with V4L backend?
Thanks!

[MartyG-RealSense]

There are other cases at #9344 and #10003 where a D455 is not detected on a native (v4L) build (on a Jetson Xavier NX) but is detected on an RSUSB build.

As well as using -DFORCE_RSUSB_BACKEND=false, did you also patch the kernel with the Jetson patch script before attempting the build on your Jetson by using the command ./scripts/patch-realsense-ubuntu-L4T.sh as described in the Jetson installation instructions for a native source-code build - Building from Source using Native Backend - at the link below, please?

https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_jetson.md#building-from-source-using-native-backend

[MartyG-RealSense]

Hi @yzkzero Do you require further assistance with this case, please? Thanks!

[yzkzero]

There are other cases at #9344 and #10003 where a D455 is not detected on a native (v4L) build (on a Jetson Xavier NX) but is detected on an RSUSB build.

As well as using -DFORCE_RSUSB_BACKEND=false, did you also patch the kernel with the Jetson patch script before attempting the build on your Jetson by using the command ./scripts/patch-realsense-ubuntu-L4T.sh as described in the Jetson installation instructions for a native source-code build - Building from Source using Native Backend - at the link below, please?

https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_jetson.md#building-from-source-using-native-backend

Hi MartyG,

I'm using NXP ARM64 based board, not Jetson series.
I mentioned Jetson nano because we found the sporadic libusb segmentation fault also happens on Jetson Nano. So I think using libusb is not as stable as native backend. Stability is quite important to us and that's why we insist to use native backend.

[MartyG-RealSense]

My research found a couple of other cases where iMX8 was being used and there were camera detection problems (with D435i in those particular cases), and an RSUSB build also corrected the problem there. So the conflict with D455 may be if you are using a linux-imx kernel and it is not supporting D455.

[yzkzero]

Thanks for the information.

The difference between 415/435 and 435i/455 is the IMU module. So it might be the root cause for the native backend detection issue. I will try to look into the source code and see if there's any findings...

[yzkzero]

I just borrowed one D435i and did the test. D435i cannot be found by native backend.

Now I believe the issue is relevant to IMU module as per my test result: D415/435 can be detected and D435i/D455 cannot. The only difference between D435 and D435i is the IMU.
So based on this analysis, can your developer make some more investigation? Or do you think it is possible to disable the IMU from HW or FW level?

Thanks!

[MartyG-RealSense]

A difference between the models that are equipped with an IMU and the IMU-less ones is that the IMU-equipped ones are treated as HID devices. This has caused problems on some platforms in the past (for example, Macs were unable to use IMU-equipped cameras such as D435i and L515 until SDK 2.42.0).

There is a case from the past week at IntelRealSense/realsense-ros#2242 (comment) where a RealSense ROS user was able to successfully use V4L with a D450 (the caseless Depth Module Kit version of the D455 which also has an IMU).

At the moment, a kernel related issue does seem to be the most likely cause of your problem if you are using a linux-imx kernel. It is likely that the developers would recommend using RSUSB since use of iMX8 with librealsense is so rare and therefore would likely be a low development priority.

I cannot think of any way to disable detection of the IMU on your D455, unfortunately, as the firmware driver identifies the camera model based on the physical sensor components that it can detect as being present.

[yzkzero]

A difference between the models that are equipped with an IMU and the IMU-less ones is that the IMU-equipped ones are treated as HID devices. This has caused problems on some platforms in the past (for example, Macs were unable to use IMU-equipped cameras such as D435i and L515 until SDK 2.42.0).

There is a case from the past week at IntelRealSense/realsense-ros#2242 (comment) where a RealSense ROS user was able to successfully use V4L with a D450 (the caseless Depth Module Kit version of the D455 which also has an IMU).

At the moment, a kernel related issue does seem to be the most likely cause of your problem if you are using a linux-imx kernel. It is likely that the developers would recommend using RSUSB since use of iMX8 with librealsense is so rare and therefore would likely be a low development priority.

I cannot think of any way to disable detection of the IMU on your D455, unfortunately, as the firmware driver identifies the camera model based on the physical sensor components that it can detect as being present.

Thanks a lot for sharing the information.
We will look into the kernel.

[yzkzero]

@MartyG-RealSense There's a good news! I noticed the libusb has been updated to 1.0.25 recently. And using this new version, all the libusb segmentation faults do not occur anymore. So we can use libusb backend solution now.

[MartyG-RealSense]

That's excellent news, @yzkzero - thanks very much for the update :)

[yzkzero]

That's excellent news, @yzkzero - thanks very much for the update :)

Hi MartyG,

Although we now can use libuse for our product. But we are still investigating using native backend as backup in case libusb has any other issue we haven't known.
So before closing this issue. Can you tell me what needs to do in linux kernel to support using V4L for D455 in RS SDK? E.g. any kernel configuration or kernel module to select?
Any information or hints will be helpful. Thanks a lot!

[MartyG-RealSense]

Sometimes one particular kernel version may be more stable than another for unknown reasons. For example, a RealSense user who was recently experiencing regular Ubuntu OS freezes found that they stopped if they used kernel 5.11.0-34-generic instead of the newer 5.13.0-27-generic kernel.

This may be related to librealsense having support for kernel 5.11 since librealsense 2.50.0 but not yet officially supporting 5.13. Unsupported kernels can work with librealsense but there may be unpredictable consequences in regard to stability.

[MartyG-RealSense]

Hi @yzkzero Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.