-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Is it possible to detect an empty shelf with depth detection or some feature of D435i? #5607
Comments
The D435i is an RGB-D camera, providing real-time point-cloud data. It can be used as a component in robotics, warehouse-automation and measurement products. There is no "detect-shelf" function in the SDK, these types of queries are too generic, with every OEM having its own set of constraints. The SDK is only offering access to camera capabilities (like point-cloud and other types of 3D data) |
@suraneti A way to detect an empty shelf may be to put image-based tags (also known as fiducial markers) at the back of the shelf that will normally be covered by products on the shelf. If a camera can see and register the tag when scanning the shelf then this would indicate that the shelf is empty. The T265 Tracking Camera can detect a type of tag called Apriltags. If you would like to continue using the D435i in your project, an option may be an 'AR marker' called Aruco. In the link below is an example of an Aruco program designed to work with the D435. |
@dorodnic, @MartyG-RealSense thanks for the commend, can you please guide me. Because I've brought D435i, so I need to find some way to use it to detect an empty shelf. If I install a D435i camera in front of a little shelf and detect from a depth changing in each step of a shelf, like full of items will be the blue color of depth and empty will be orange/red, if orange more than 70% then alert. Can this idea work though python wrapper? |
The first step of the idea that you have in mind may be to get a colorized depth image, like in the Python code in the tutorial linked to below. The part about detecting the color percentage (or the 'dominant color', which is what you really need) is more complex and may involve incorporating vision software such as OpenCV into the project. I got some useful leads by googling for 'dominant color python''. Such as these examples: https://stackoverflow.com/questions/52398237/how-to-find-the-dominant-color-in-images-using-python https://adamspannbauer.github.io/2018/03/02/app-icon-dominant-colors/ |
Can you tell me how are you planning to get the image? I mean is the camera mounted somewhere at a constant static location like rooftop etc or it is supposed to be attached to some moving vehicle/robot etc.? As a suggested solution would need this information. |
@kafan1986 Thanks for your response! As suraneti said "If I install a D435i camera in front of a little shelf", I inferred from this that the camera will be static and not on a mobile shelf scanning robot. |
Colourizing depth is just for visualizing. You can work with direct depth values. Your approach is OK. But as I have done some work with the D415 and other depth cameras. Some of the issues that you will likely face: So in all you will be engaged with tweaking some threshold values and it will always be a fight between reducing false positives and false negatives. |
@MartyG-RealSense @kafan1986 first of all, thanks for your support. I will try with "dominant color" and background subtraction. I will update the progress. P.S. I've found a empty shelf detection project only camera with object detection, should I go this way better? |
I think you intended to link to this project by the same developer? https://github.com/thom1178/supermarket-shelf-detection Using Deep Neural Networks (DNN) for this purpose may be greater complexity than the task needs, compared to doing it with methods such as dominant color. If you are prepared to go to the effort of using a DNN then there's no reason not to do it if you are happy to make that effort and it provides the result that you need. Using DNN for empty shelf detection may be good practice if you plan to expand the scope of the project at some future point to use machine learning to perform tasks such as recognizing and checking the stock items left remaining on the shelves. An example of such shelf checking is the mobile retail robot Tally, which can navigate a store and scan the shelves. Tally uses a combination of RealSense depth sensor and Lidar sensor. |
A website very recently demonstrated 3D maze navigation (similar in concept to a retail store layout) using a RealSense D435 instead of lidar though. |
@suraneti My experience taught me in variable conditions where product will be varying, light conditions will be varying as well as position of products. A only deep learning project won't achieve more than 80-85% accuracy and that too this is best case scenario. simberobotics link shared by @MartyG-RealSense seems to use multiple data points to work with like depth, computer vision and RFID. It is really difficult and sometimes not possible to achieve high accuracy without using sensor based data. For your use case a Lidar based moving robot, which can move around the store and can use a single lidar sensor and move the sensor vertically at different adjusted heights to match the different shelf level and then compare the depth to know empty product position, probably be a good approach and then adding a additional layer of computer vision based deep learning might increase additional accuracy. Also, you have IMU data that help with creating pre-layout of the store and that will help navigation in store etc. I am just throwing in some ideas, all the best to you. |
Issue Description
I've searched an documentation or code example but no document related to my ongoing empty shelf detection project.
The text was updated successfully, but these errors were encountered: