Counting “Incomplete video frame detected!” occurrences to detect hardware problem

[pwalczykk]

Required Info
Camera Model	D415
Firmware Version	05.09.14.00
Operating System & Version	Ubuntu 16.04
Kernel Version (Linux Only)	4.4.0-131-generic
Platform	PC
SDK Version	2.16.0
Language	C++, Python
Segment	Robot

Hi everyone,

We are using 4 Realsense D415 cameras in our robotics arm setup. They are connected to 4 independent USB 3.0 controllers on PCI expansion card. Overall this system is working reliable, but from time to time one of USB connections degrades. This is typically caused by robot’s movement that bends the cable over and over.

While using low quality or degraded cable with realsense-viewer we get “Incomplete video frame detected!” notification much more often than with new and professional cables. This pushes us to idea of monitoring cable degradation process by counting amount of incomplete frames received in time. It seems that rs-fw-logger does provide proper log information every time some incomplete video frame is received, but I am unable to decode and distinguish it from another logs.

Can you provide a way to determine if rs-fw-logger log was related to incomplete frame, and for which camera_id this log was created?

Is there any other way I can access incomplete video frames information while using realsense_ros_camera nodelet for streaming camera data?

[dorodnic]

Hi @pwalczykk
Using librealsense, you could register to sensor notifications to receive callback when they occur.
To access incomplete frames you'd need to do some modifications to librealsense.
We can assist if this is something you'd like to do.

[pwalczykk]

Thanks @dorodnic, this can not be my top priority, but I would still like to do it. Any help will be appreciated. Solution that satisfy me the most would be adding ROS publisher that broadcast message on every incomplete frame. I made a quick check in librealsense and found "Incomplete video frame detected!" message in v4l_uvc_device::poll() from backend-v4l2.cpp. I wonder if backend-v4l2 is used by realsesne_ros_camera or is there any equivalent? Or maybe you have some other suggestions where should I start?

[dorodnic]

I can ask ros-realsense maintainer @doronhi to add these notifications to our wrapper, but it can take some time.
Basically, all you need for detecting these events is:

for (auto&& s : dev.query_sensors())
{
    s.set_notifications_callback([&, dev_descriptor](const notification& n)
    {
          if (n.get_category() == RS2_NOTIFICATION_CATEGORY_FRAME_CORRUPTED)
          {
               std::cout << "Detected partial frame!" << std::endl;  
          }
    });
}

Where dev is either the one you get from pipeline.start().get_device() or from context().query_devices().

If you want to get these partial frames data, I think it should be sufficient to comment out backend-v4l2.cpp:959-970
This will cause some artifacts, since pixels from older frames will be used to fill in the gaps, so you probably would also want to reset frames to black before copying the partial content. sensor.cpp:490 is probably the right place.
Then just build librealsense from source (cmake . && make && sudo make install) and you should be good to go.

[pwalczykk]

I will try to add ROS notifications on my own this week. In the mean time, I would be really glad if you could estimate order of magnitude of time that it can take you to add this functionality.

[pwalczykk]

Ok, I got it working. Updated realsense_ros_camera node publishes amount of incomplete frames detected every time it detects frame drop. For each camera separately. Thanks for help @dorodnic!

        // Setting up notification callback that counts incomplete image frames received from realsense camera
        // On every incomplete frame, atomic counter is increased and message with new counter value is published
        for (auto&& s : _dev.query_sensors())
        {
            s.set_notifications_callback([this](const rs2::notification& n)
            {
                if (n.get_category() == RS2_NOTIFICATION_CATEGORY_FRAME_CORRUPTED)
                {
                    std_msgs::UInt64 msg;
                    msg.data = ++this->_incomplete_frames_counter;
                    this->_incomplete_frames_publisher.publish(msg);
                }
            });
        }

[dorodnic]

Awesome :) Always happy to help
@doronhi - please consider doing something similar in one of your future releases.

[pwalczykk]

Perfect! I will let you know if we have some interesting conclusions, though our experiments will probably take few weeks. I can also make a PR when our feature reach some final shape.