-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to calculate the world coordinates for a pixel point with a depth value of 0? #12304
Comments
Hi @OldLiu666 If your depth problem is primarily due to reflections then applying a physical optical filter prodct called a linear polarization filter over the lenses on the outside of the camera can significantly help to negate glare from reflections. Any thin-film polarizing filter should work so long as it is linear, so they can be obtained inexpensively. You can search stores such as Amazon for the term for the term linear polarizing filter sheet to research example filter products. Section 4.4 When to use polarizers and waveplates in Intel's white-paper guide to using optical filters with RealSense 400 Series cameras has more information about this type of filter. The image below from that section demonstrates the before and after effects of a linear polarization filter. The means of attaching the filter externally to the outside of the camera does not need to be advanced. It could be as simple as wrapping elastic bands round the camera to hold the filter firmly onto the front. Alternatively, you could try setting the Medium Density camera configuration preset to see if that improves your depth image results, as Medium Density provides a good balance between accuracy and the amount of detail on the depth image. Python code for setting the Medium Density preset can be found at #10695 (comment) whilst the link below provides C++ code for setting a named preset. |
Hello and good day! First of all, I'd like to express my gratitude for your diligent answers. Perhaps I didn't clarify earlier. What I meant was that one of the possible reasons for the depth value of a point in the depth map being 0 could be due to reflection, among other reasons. My main emphasis is on the missing depth data. I mainly use the depth data measured by D435 and then calculate the three-dimensional coordinates of the pixel using the rs.rs2_deproject_pixel_to_point function, which is then combined with the millimeter-wave back-projection algorithm (BP algorithm) for imaging. Therefore, the depth and coordinates of the pixel are crucial. Missing depth information can lead to an inability to calculate the three-dimensional world coordinates, which in turn affects the quality of the imaging. |
Hello and good day! First of all, I'd like to express my gratitude for your diligent answers. Perhaps I didn't clarify earlier. What I meant was that one of the possible reasons for the depth value of a point in the depth map being 0 could be due to reflection, among other reasons. My main emphasis is on the missing depth data. I mainly use the depth data measured by D435 and then calculate the three-dimensional coordinates of the pixel using the rs.rs2_deproject_pixel_to_point function, which is then combined with the millimeter-wave back-projection algorithm (BP algorithm) for imaging. Therefore, the depth and coordinates of the pixel are crucial. Missing depth information can lead to an inability to calculate the three-dimensional world coordinates, which in turn affects the quality of the imaging. |
You are very welcome! Please let me know whether using the Medium Density Visual Preset suggested above improves the depth image. There was a RealSense-equipped wheelchair at the link below that combined RealSense with a millimeter-wave radar sensor. https://www.intelrealsense.com/smart-wheelchairs-with-luci/ A RealSense C++ tool that sounds similar to the map fusion involved in millimeter-wave is the rs-kinfu project. It uses a single RealSense camera to progressively build up a pointcloud over time via frame fusion as the camera is moved around. https://github.com/IntelRealSense/librealsense/tree/master/wrappers/opencv/kinfu I have not heard of the iron-ball calibration method and do not have RealSense-related information about it, unfortunately. |
ok,thank you very much!!! |
Hi @OldLiu666 Do you require further assistance with this case, please? Thanks! |
Hello again, sorry to bother you. I'm looking to perform 3D reconstruction using RGB-D images captured with the D435 and combined with the TSDF (Truncated Signed Distance Function). I've found that there's limited code available on GitHub for this purpose. Do you know how to capture photos with the D435 and then proceed with TSDF-based 3D reconstruction? Or perhaps, capture a series of depth images first and then perform the reconstruction? |
If might be possible to use your D435 with the PCL-based Kinfu Large Scale project at the link below (not the RealSense example rs-kinfu) to achieve your goal by converting the TSDF data into a mesh. |
hello, thanks again! The challenge I'm facing is not converting the TSDF data into a mesh. Instead, it's about generating a TSDF grid using RGB-D images captured with the D435 and integrating it with the Open3D library. Once that's accomplished, I plan to proceed with tasks using the TSDF. |
I am not personally familiar with TSDF grids, but my research of your question suggests that a PDF document of a research paper at the link below may be a useful reference. The paper is titled 3D Scene Reconstruction from RGB-Depth Images. https://fkong7.github.io/data/CS284_Final_paper.pdf Section 1.2 - Mesh Reconstruction - on page 2 of the paper discusses using a RealSense camera (which appears to be a D435 or D435i) to do the following. We applied the TSDF algorithm to reconstruct a 3D voxel grid from multiple input frames. With the estimated camera |
I sincerely thank you once again for your assistance. I took a brief look at the article, and it involves TSDF 3D reconstruction using RGB-D images captured with the Intel RealSense camera, which aligns with what I'm looking for. I will read the article in detail, hoping it proves beneficial. Lastly, I genuinely appreciate your tireless efforts in addressing my queries and providing solutions. Thank you.Wishing you success in your work and a joyful life. |
You are very welcome. I'm pleased that I could be of help. Thanks for your kind words! |
|
Hi @OldLiu666 The D435 camera is not equipped with an IMU component to detect the angle that the camera is oriented at. RealSense 400 Series cameras also cannot provide 'pose stream' data (the position and angle of the camera) as that feature is only on the RealSense T265 Tracking Camera model. It may be possible to establish a pose for D435 by mounting both a D435 and a T265 on a bracket and moving the bracket, since the T265's orientation will also be the D435's orientation. Another approach for RealSense cameras without an IMU is to establish the camera's angle using a plane-fit algorithm, as described at the link below. https://support.intelrealsense.com/hc/en-us/community/posts/360050894154/comments/360013322694 |
I apologize for not providing the complete information; my camera is the D435i. But as you mentioned, "RealSense 400 Series cameras also cannot provide 'pose stream' data (the position and angle of the camera)." Does this mean I have to find a T265 or use another algorithm, such as the plane-fit algorithm, to determine the camera's pose? |
It is possible to obtain the camera tilt / roll angle from a D435i. The camera does not know its own position though. Only a T265 can provide that information with its pose stream. https://dev.intelrealsense.com/docs/rs-pose Below are examples of D435i-compatible IMU programs. C++ Python |
Thank you for your efforts in providing the solution. I wish you a pleasant life. |
Case closed due to solution achieved and no further comments received. |
After completing the alignment of the depth stream and color stream, we can then obtain the depth data of each pixel, and then use the function rs.rs2_deproject_pixel_to_point(depth_intrin, [i, j], depth_value) to calculate the world coordinates of the pixel. However, for the depth map, there are always some pixels that have a depth value of 0 due to certain reasons, such as reflection, as shown in the black parts of the image below. The world coordinates calculated through the function rs.rs2_deproject_pixel_to_point(depth_intrin, [i, j], depth_value) for these pixels are (0,0,0). I want to know if there is any method to calculate the three-dimensional coordinates of all pixels. I have used hole_filling = rs.hole_filling_filter() on the depth map, but the results are not ideal. Some pixels have a depth value less than one, while some others suddenly become very large, greater than ten, which is not conducive for use.
The text was updated successfully, but these errors were encountered: