D415 depth sensor streams seem to have stopped working - defective device?

[puzzlepaint]

Required Info
Camera Model	D415
Firmware Version	05.13.00.50
Operating System & Version	Ubuntu 20.04
Kernel Version (Linux Only)	5.15.0-46-generic
Platform	PC
SDK Version	2.50.0

Issue Description

I have 32 D415 cameras here in a volumetric capture setup, and unfortunately it seems that for one of them, the depth sensor streams (depth or infrared images) have stopped working. I believe that no physical harm has happened to the device. I can rule out the cable or other external factors since I connected it with a cable to a PC with which other cameras work fine.

The color stream of the device still works. I use the RSUSB backend and interestingly, by default the color stream was very choppy this way, and it was constantly spamming messages about the device getting re-attached to the syslog. I then reverted commit #9821 in my local checkout of librealsense since I had read that this caused such issues in #10174. This fixed the choppy stream. I did not observe the same level of choppiness with other devices before. I don't know whether it has already been as choppy before the depth sensor stopped working, or whether it was caused by that.

There don't seem to be any unusual errors or warnings in realsense-viewer when I try to start the depth stream on the device. It does log some errors and warnings, but I see the same log messages when starting the depth stream on a working device, so I think that they are not relevant. The only specific message seems to be types.h:898 - Depth stream start failure / notifications.cpp:511 - Depth stream start failure. In the GUI, realsense-viewer keeps showing "No Frames Received!"

Now I wonder whether there is any way to repair the device. Things that I have tried unsuccessfully so far:

• Trying to stream depth images on the lowest resolution to check whether depth streams work at all (they don't)
• Performing a firmware upgrade
• Resetting the dynamic calibration parameters with Intel.Realsense.CustomRW.exe -g

Is there anything else that could be worth trying?

[MartyG-RealSense]

Hi @puzzlepaint The kernel version 5.15 should not be an issue, as you are using the RSUSB backend and so the kernel will be bypassed by the RealSense SDK.

Are the cameras used one at a time or enabled in multiple quantities simultaneously by a single program? The RSUSB backend is not the best choice of build for applications that control multiple cameras from the same program as it is suited to single-camera applications, with a patch-based build of the SDK (V4l / native backend) being the best option for multicam projects.

Whilst the SDK does not currently include support for kernel 5.15 in version 2.50.0, an unofficial test patch to add support for kernels 5.13 and 5.15 is available at #10439 (comment)

Also, are all 32 cameras attached to the same computer?

[puzzlepaint]

Are the cameras used one at a time or enabled in multiple quantities simultaneously by a single program?

Each camera is attached to its own computer. So, possible multi-camera issues can also be ruled out completely. I am sorry that my initial description was unclear in that regard. I only mentioned the 32 cameras in total to say that I am sure that the issue is a device-specific problem for that single device that seems to be defective, since all of the others work fine in the same type of setup.

Is there anything that could be attempted to repair the single defective device whose depth sensor does not start anymore?

[MartyG-RealSense]

If you do not mind invalidating the sales warranty on that particular camera then you could open its casing by detaching the USB cable from the side of the camera to reveal a single small screw (inbetween 'CE' and 'FCC'). Once the casing is open, you could then check whether the interposer cable connecting the Vision Processor D4 Board and the D415 Depth Module board is fully seated in the 50 pin connectors at both ends of the interposer cable.

Alternatively, you could perhaps alter the spacing between other cameras so that their fields of view overlap the area that the affected camera was observing, therefore enabling that camera to be retired from your volumetric system.

[puzzlepaint]

Thank you for your reply. Since I believe that there is no warranty on the affected device anymore, I will try to open it and check the cable you described. However, I will need to find a suitable screwdriver first, since so far the case screw did not seem to move when I tried to use the tools on it that I have available.

[puzzlepaint]

It seems that now a second device broke. It shows almost exactly the same behavior as the first one: The depth sensor (depth or IR stream) does not start anymore, while the color sensor still works. There are some additional items in the log in realsense-viewer when trying to start a depth stream on this device:

This is printed frequently, seemingly each time types.h:898 - Depth stream start failure is printed:

types.h:898 - REC error

This is printed occasionally:

types.h:898 - Right MIPI error

I wasn't able to open the first device yet, but for this second device, it seems unlikely to me that any cable has become loose (at least due to physical shocks), as it was just mounted in a fixed position where it definitely worked before.

Are there by any chance other probable guesses as to what could have made these devices break? The things that I can think of right now are:

• The humidity is slightly higher than usual here, with usual conditions of around 70% relative humidity at about 20 degrees Celsius temperature. Is that too much or should this be acceptable? I looked into the datasheet for the D400 cameras and it mentions 40 degrees Celsius and 90% relative humidity, but I am not sure how that transfers to 20 degrees Celsius.
• I use the cameras on active USB hubs because they are attached to Raspberry Pi 4 devices, which cannot supply enough power on their own to both the cameras and the SSDs that I attached to them as well. It seems that both faulty devices were plugged in such that when powering on the system, the active USB hub was powered on before the Raspberry Pi 4 that it is attached to. Is this by any chance known to be able to break the cameras if the USB hubs somehow don't behave properly?

This is unfortunately a bit of a nightmare scenario for me, as I invested a lot of time and money into the hardware and software of this volumetric video system. Perhaps the two broken cameras in a row were a coincidence now, but who knows. I would be very happy about any effective suggestions on how to prevent further cameras from breaking - thanks!

[puzzlepaint]

I found this analogous case where the depth sensor on a camera stopped working with the "Right MIPI error" log message: #9947
Unfortunately, it seems that this camera needed to be returned to the seller. It would be important to know whether this breakage can be avoided.

[MartyG-RealSense]

Camera hardware should not be harmed if insufficient power is supplied to them by the USB system. They would either not be detected or fail to activate a particular camera modeif the camera tries to draw upon the power required to enable that mode and insufficient power is available.

If your computing setup can make use of ethernet then you could explore the use of PoE (Power over Ethernet** "hats" for your Pi's to provide their power needs.

https://www.raspberrypi.com/products/poe-hat/

The Right MIPI Error is not likely to be a significant concern if it only occurs oocasionally.

---

In the data sheet description of humidity, RH stands for Relative Humidity, which is "a measure of how much water vapor is in a water-air mixture compared to the maximum amount possible". Normal humidity is considered to be between 30 and 50 percent. Anything under 30% is considered too dry. Equally, anything above 50% (like your 70% humidity) may be too wet.

The Electronics section of the Wikipedia article on humidity explains how effects such as condensation at high humidity can cause short-circuits and may permanently damage electronics if the condensation has not been given an opportunity to evaporate before activating the device.

The data sheet's Section 4.6 that features the humidity recommendations provides a maximum of 90% relative humidity, it states though in the footnotes beneath the table that the camera's performance may be negatively impacted by extended exposure to excessive humidity.

https://en.wikipedia.org/wiki/Humidity#Electronics

Would it be possible to provide additional heating in the location that the cameras are used in to help to prevent condensation from forming inside the electronics? Either that or individually warm the cameras up externally by applying heat to their outer casings to evaporate condensation before powering them on.

[puzzlepaint]

Thanks for the extensive reply!

I swapped the power plugs anyway in order to ensure that each Raspberry Pi and their active USB hub will be powered on at the same time, to prevent a temporary lack of power during startup that could possibly lead to other issues.

Later, I happened to notice that the LEDs on some of the Raspberry Pis were regularly flashing up briefly, roughly every 10 seconds, even though the corresponding switch on the power strip that they were connected to was off. This seemed very odd to me at first, but apparently it has happened similarly to others as well. Seems like the Raspberry Pi's official power supply somehow manages to charge up over time when connected to some kinds of power strips that are switched off, and supply some power from time to time, causing the flashing LEDs.

Now I wonder whether this could be a likely cause of the broken cameras. (Unfortunately, due to swapping the power plugs, I am not sure whether it affected exactly those devices that broke.) Supposedly, the cameras received some power briefly every ~10 seconds. Could it be that this has worn them out over time? The Raspberry Pis themselves and their SSDs seem to have survived it for now, however.

---

Constantly heating up the room would be doable in principle, however I would prefer to only do that as a last resort due to the associated energy consumption. I am not sure how likely it is that the humidity is the issue. I guess that the humidity should in principle affect all of the electronic devices at the location, but none of the other equipment has had any failure yet. Are the cameras more susceptible to it than other devices?

[MartyG-RealSense]

Humidity is not typically an issue that is raised by RealSense users unless there is a specific situation where humidity is known to be a risk factor in the location where the camera will be placed (such as a stationary position in an outdoor location with a damp climate). I would think that any electronic device with an open ventilation grille would be vulnerable in such environmental conditions though.

It does not seem likely that the Pi receiving power every 10 seconds would cause damage to the camera attached to it.

[puzzlepaint]

Thanks again for your reply. Then I guess I'm out of ideas for what the likely cause was. Maybe it was indeed just two cases of bad luck.

I did get a suitable screwdriver now and disassembled one of the broken devices as far as the disassembly steps were clear to me. The broad black cable seemed to be firmly in place in the green board. It seems like the other side may be less easy to access? At this point I'm not sure how to open it up further.

[MartyG-RealSense]

If the interposer cable joining the depth and D4 boards was firmly attached to the connectors at both ends then further disassembly of the cameras will be unnecessary .

[puzzlepaint]

I can't reach the connector on the end that is still attached to the part with the cameras in the photo above. However, looking at it from the outside, I guess that it is probably fine.

[MartyG-RealSense]

And resetting the calibration in the Viewer using the instructions at #10182 (comment) makes no difference to the depth stream start failure errors?

[puzzlepaint]

Resetting the calibration in this way does not seem to work in my case. The "Write Table" button only seems to be active while depth sensor streaming is currently active. Under this condition, the following errors occur:

• Clicking "Restore Factory" results in:

  close() failed. UVC device is streaming!
  

• Clicking "Write Table" results in:

  hwmon command 0x51( 0 0 0 cafecafe ) failed (response -1= Invalid Command)
  

I tried this on a dual-boot Windows installation with the most recently released official RealSenseViewer executable to ensure that my specific configuration on Linux with the RSUSB backend does not cause issues.

As mentioned in the first post, I also tried resetting the configuration with Intel.Realsense.CustomRW.exe -g, which prints that it did reset the calibration on the device successfully. But this does not make the depth stream work.

[MartyG-RealSense]

Have you tried the affected cameras on one of the other 30 Pi boards to confirm whether the problem is with the particular Pi boards that the cameras with the problem are attached to rather than the cameras themselves?

Going through this case from the beginning, I took note of this quote:

It seems that both faulty devices were plugged in such that when powering on the system, the active USB hub was powered on before the Raspberry Pi 4 that it is attached to.

This reminded me of past issues, particularly with ROS, where the camera would not work correctly when used with a USB hub because the hub received power first before the computer / computing device had.

I have seen a couple of cases where RealSense users opted to use PCI/E expansion cards with USB ports instead of a hub, like the example for Pi 4 at the link below.

https://www.amazon.com/USB-3-2-Expansion-Gen1-PCIe/dp/B092VVDPWX

[puzzlepaint]

I tried the affected cameras on my desktop PC, on which other cameras work fine. So I am sure that the problem is with the cameras and not with the Raspberry Pis.

I already changed the cabling such that the USB hubs won't receive power before the Raspberry Pis that they are attached to anymore, so that shouldn't be an issue anymore in the future. In any case, I guess that this is not supposed to damage the cameras permanently.

The linked expansion card seems to be designed for the RPi 4 Compute Module, which as far as I remember has a usable PCIe port, unlike the standard Raspberry Pi 4.

---

Seeing that there does not appear to be a clear indication about what the likely cause of the damage was, and there also does not seem to be a way to repair the affected cameras, I would say that this ticket can be closed for now. I might re-open it in case another camera breaks and it gives any new clue about the cause of the problem (but let's hope that this won't happen ...). Thank you for your assistance so far!

[MartyG-RealSense]

You are very welcome. :) Good luck!