-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Encountered a problem when using the mask prediction module as a preprocessing module for other algorithms #11
Comments
Hi, Thanks for your interest! Unfortunately, we cannot release the code of this part right now, but I may try my best to help you find the problem. Could you please provide more details about your implementation, such as the dataset you used, how you trained the mask prediction module, and which fully overlapping point cloud registration method you used? |
Hi, I am very grateful for your quick and enthusiastic response. Let me explain my current situation: Firstly, I am using the Feature-metric Registration (FMR) method for point cloud registration, which is a method for fully overlapping point cloud registration. Secondly, I am using the Modelnet40 dataset, and my strategy for generating partially overlapping point clouds is inspired by the way RPMNet generates partially overlapping point cloud pairs. Then, I have noticed that OMNet consists of four modules: encoder, fusion, decoder, and regression. I have only incorporated the encoder, fusion, and decoder into the training process of FMR. Since I am worry about the accuracy of the pred_mask from the untrained decoder, I first trained the mask prediction module (encoder + fusion + decoder) separately for 20 epochs. During this process, no transformation matrix was generated, so I couldn't change xyz_src_iter at the end of each training iteration as OMNet does. After 20 epochs, I integrated the training of the mask prediction module with the FMR network: the input point clouds p0 and p1 are first passed through the mask prediction module, and then I use the pred_mask output by the decoder of OMNet on them (I thought that this way they would become point clouds of the same shape, and then I replaced the points whose coordinates became (0,0,0) with the points whose coordinates are not (0,0,0)). After that, I fed them into the FMR network for training. |
I see. So here are my suggestions,
|
Your suggestions are very helpful, I will give them a try. But I still have some questions regarding the third suggestion. |
If I remember correctly, I trained them separately. |
Thank you very much for your reply. I have made some attempts in the past two days but have failed. |
I see. Actually, it may not be proper to directly regress the rotation matrix since the degree of freedom is three not nine (3*3 rotation matrix). There are some packages that provide easy ways to convert the quaternion to a rotation matrix, such as Pytorch3D. Hope it is helpful to solve your problem. |
I'm sorry to bother you again after such a long time, but I do need to ask you a question. |
Hi, If I understand your problem correctly, you can obtain a higher accuracy of the overlapping mask prediction using my code but cannot obtain a normal result when changing to your own code and data. I am not sure about the reason since I do not have your code and data. Also, it may take me a long time to help you check your code. For your confusion about the data preprocessing during training, it is easy to find the answer from my code. When doing the evaluation, there is a flag named "deterministic" will be turned on and the data will be fixed. Note that this flag is turned off during training, which means the data augmentation is random. So, I recommend you modify my code to meet your requirements directly. This may reduce the probability of writing bugs. |
I'm sorry to bother you again, but I encountered some issues during the visualization process while using your code. |
If your mask prediction accuracy is around 90% as you mentioned before, you should obtain similar shapes after applying the masks to the input point clouds. For Fig. 5, we show the predicted mask after applying argmax, and the predicted overlapping and non-overlapping points are shown in red and blue, respectively. As for the error map visualization of the global features, please refer to the description in Sec. 5.1. |
I'm sorry to bother you again,I encountered a very confusing problem. During training, I set "transform_type" to "modelnet_os_rpmnet_clean". After training for 1000 epochs, I loaded the model saved during the training process as "self_model_latest.pth". Then, I generated a new set of data using the same processing method as the training set. However, the performance of predicting overlapping regions using the model on this new set of data was very unsatisfactory. The following is the output at a certain moment during the training process: Here is the output from a certain test process: I can't understand why the performance is good during training, but it is still poor when testing on the training dataset. Another thing is that there is an error when executing "model.load_state_dict(state["state_dict"])". Therefore, I use "Using custom loading net" instead. Could this be the reason for the problem? Looking forward to your reply. |
Hi, Maybe you should check if the pre-trained weights are loaded successfully. |
Hi, your suggestion is correct and it helped me solve the problem. |
Hello, thank you very much for providing such outstanding work. I noticed that you once said, 'The mask prediction module in this method can be used as a preprocessing module for other point cloud registration methods, although these registration methods may not be designed for partially overlapping point cloud registration.' You have also successfully combined the mask prediction module with methods such as PointNetLK, DeepGMR, and DCP, and achieved significant results. I have been trying to combine the mask prediction module with a method for fully overlapping point cloud registration for a month, but it has not been successful yet. Therefore, I would like to request that you provide me with the code to combine the mask prediction module with other methods, so that I can learn the specific details of use. My email is [email protected]. Thank you very much!
The text was updated successfully, but these errors were encountered: