Using 5 D450 cameras, libusb "cannot allocate memory" #2242
Comments
|
Hi @v-lopez You have a powerful 11th generation processor that should be capable of supporting 5 cameras simultaneously (and can do at lower settings or reduced number of topics, as you described above). Are your cameras attached to a USB hub? If they are then you may be using up the available data bandwidth on the hub when running all 5 cameras with multiple topics or a higher resolution / FPS. In Intel's multiple camera white-paper document, they published tables estimating the bandwidth usage in Mbps for up to 6 cameras in different streaming configurations when attached directly to a PC or to a hub. If you are using a hub and the maximum available data bandwidth on the hub is being exceeded, you could consider adding a second hub and putting some of the cameras on it. The USB standard allows 5 hubs to be linked together. I recommend aiming to not exceed 5000 Mbps/ 5 Gbps total bandwidth consumption on a single hub. |
|
Hi @MartyG-RealSense , thanks for the quick response. The PC is this: https://download.asrock.com/IPC/Download/e-catalog/NUC%20BOX-1135G7.pdf I am not using any Hub, I am using 2 cameras through USB-C to USB-C cables, and 3 cameras through USB-C to USB-A. Using all the front and back USB ports of the NUC. |
|
If all the cameras are attached directly to the computer's USB ports then they will be drawing all of their power from the computer's power supply unit (PSU), and the stability of power supply to the USB port may be less than that of a mains electricity powered USB hub. I recommend budgeting up to 2 watts power consumption for each attached camera. If you are using your own choice of cable instead of the official USB A to C cables then the cable should ideally be a premium quality one due to the high volume of data bandwidth that RealSense cameras can transmit through the cable. USB-C cables can have quality issues, and a lower quality cable could cause degraded performance or disconnections, especially when the cable is 2 meters or longer. |
|
The cables are USB 3.2, and 0.5m long. And it still would not explain, why disabling infra1 solves the problem. Because both infrared projectors are still running if I request depth information. I believe the problem might be related to what was commented here jetsonhacks/buildLibrealsense2Xavier#17 (comment) |
|
If you need infra1 but do not need infra2 then disabling infra2 is a valid way to save bandwidth, as described at #1220 (comment) Depth frames are generated from the left and right raw infrared frames in the 400 Series cameras at the point of capture in the camera hardware. It is not the same as streaming left and right infrared in the RealSense SDK. Because of this, you can still stream depth from a 400 Series camera with only one infra stream enabled or no infra stream at all. |
|
Unfortunately we need all of them, rgb, infra1, infra2 and pointcloud. |
|
You may be reaching the practical limits of the number of RealSense cameras that can be handled on the same computer with multiple streams from each of the 5 cameras. Even in 2018 Intel were recommending a maximum of 4 simultaneously active cameras on the same computer on an i7 machine. The more cameras that are added, the more of the computer's resources that are consumed. If adding a second PC is viable for your project, Intel have a RealSense ROS guide for dividing multiple RealSense cameras across 2 PCs and communicating between the machines with ROS_MASTER_URI If the message messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11 is generating continuously multiple times per second, as it is in your log, then it indicates a serious communication problem between the code and the device. You could try adding initial_reset:=true to your roslaunch instruction to reset the camera at launch to see whether it reduces the occurrence of the control_transfer returned messages. |
|
I tried with initial_reset:=true without luck. I tried with the 5 cameras on separate nodes, as well as all on the same node. The total CPU usage is always below 200% (this machine has 8 cores, so the max would be 800%). The computer should be able to handle much more CPU/memory wise. I think it's related to the libusb, but I have no clue as to how it's used. Edit: Another PC is not a viable solution unfortunately. |
|
If you build the librealsense SDK from source code with CMake and include the build flag -DBUILD_WITH_OPENMP=true then YUY to RGB conversion and depth-color alignment operations can take advantage of multiple cores, reducing latency at the expense of greater CPU utilization. Your log states that you have align_depth = false though, so you may not get a noticable performance advantage from using this method. If your NUC has Windows installed as the asrock specification suggests then librealsense would be using Windows Media Foundation as its backend by default. If the CMake build term -DFORCE_RSUSB_BACKEND=true is used then the SDK build on Windows should use a backend based on a WinUSB rewrite of the UVC protocol instead. |
|
Thanks I'll try the OPENMP thing. |
|
I tried with initial_reset:=true without luck. I tried with the 5 cameras on separate nodes, as well as all on the same node. The total CPU usage is always below 200% (this machine has 8 cores, so the max would be 800%). The computer should be able to handle much more CPU/memory wise. I think it's related to the libusb, but I have no clue as to how it's used. |
|
RealSense 400 Series cameras and the librealsense SDK are designed to run on a very wide range of configurations, from very low spec to high spec. So unless a command to make use of multiple cores under very specific conditions is used (such as using 'make -j8' for faster compilation when building librealsense with CMake or using the flag DBUILD_WITH_OPENMP=true for multi-core support of alignment and RGB color conversion) then it is likely to just utilize a single core by default. |
|
If your project has to use a NUC due to size constraints then it is conceivable that you could add to it an external Nvidia graphics GPU (an eGPU) which takes over as the graphics chip from the inbuilt graphics chip. This could potentially enable you to take advantage of librealsense's CUDA support, which accelerates color conversion, pointcloud and align processes by offloading work from the CPU onto the GPU. eGPUs that fit onto NUCs are expensive though. https://www.intel.com/content/www/us/en/support/articles/000056230/intel-nuc.html Edit: Your NUC would need a Thunderbolt port to add an eGPU. The Razer Core X is an example of an Nvidia compatible eGPU. |
|
Thanks for those suggestions, we'll take them into account. Regarding
It doesn't help. And I have lots of these errors even with just one camera being used: |
|
Are the above errors from a camera on a USB A to C cable or a C to C cable? |
|
I just left the office and won't be able to get back until Wednesday. I'm building the realsense library with v4l to try a different backend, will try it on Wednesday as well . |
|
The suggestion is based on a small number of past reports I have seen on this forum of worse performance when using C to C cables compared to A to C. For example, the RealSense-compatible Dot3D scanning software suggests in their troubleshooting documentation link below to make sure that the chosen C-to-C cable supports 10 GBps of data transfer bandwidth. https://desk.zoho.com/portal/dotproduct/en/kb/articles/rsconnectionissues |
|
I can confirm that it happens with each camera, these errors are more common on USB-C to USB-C cables, but they do happen on USB-C to USB-A as well. What worries me, is that every now and then, one of the cameras will stop publishing PCL during 3-5 seconds. For instance, here are the logs of such an occurrence, I am printing every second the messages received in the last second, for each type of image, and the delay of those messages (current timestamp - msg stamp). This is with the cameras configured at 15Hz, and 640x360 resolution on all topics. I see the cameras are identified as using 5 GBps of bandwidth, do you have any cable recommendations? |
|
The company Newnex are a supplier of industrial grade USB cables and equipment and provide dedicated support for RealSense cameras. https://www.newnex.com/realsense-3d-camera-connectivity.php Intel test the cameras for long-running (hours and days). If the camera remains within recommended operating temperature tolerances then they should be able to run indefinitely. The official maximum operating temperature is 35 degrees C, though in practice the camera can likely go up to 42 degrees before problems start occurring beyond that temperature. If the Projector Temperature exceeds 60 degrees C for several seconds then a laser safety mechanism in the firmware driver will first try halving the laser power. If the Projector temperature continues to exceed 60 degrees after that then the laser is shut off, which also removes the IR dot pattern that the 400 Series camera can use to analyze surfaces for depth detail. The camera can continue operating whilst the laser is disabled but the depth image quality may reduce noticably if the scene that the camera is located in is not strongly lit (as the camera can alternatively use ambient light instead of the dot pattern for depth analysis). If an error does occur after a long period of time has passed during a long-run session and the camera's operating temperature is within recommended tolerances then it is typically due to a non-camera factor such as a glitch in the USB cable / USB port, a computer hardware problem or a glitch in the computer's operating system. |
|
Thanks will take a look at that cable provider. I was also able to test with V4L, and I am getting better results in all aspects. I am running 5 cameras simultaneously, with no periods missing data and I am running them at 640x480 (with the other driver I could only reach 640x360). After half an hour or so the Projector temperature for all cameras is sitting between 40-45º. So for now it seems like switching to V4L has solved all the issues I had. |
|
That's great to hear that V4L has provided a significant improvement for you, @v-lopez - thanks very much for the update! |
|
Hi @v-lopez Do you require further assistance with this case, please? Thanks! |
|
No, all good! |
I am trying to use 5 D450 cameras on the same NUC pc.
I'm attaching the full log of the 5th camera, but I believe the significant part is:
5th_realsense.txt
The text was updated successfully, but these errors were encountered: