Implement RTMDet to Perception Pipeline #7235
Comments
StepTurtle
added
the
component:perception
|
Here is the results of pre-trained models shared in this link from mmdetection. Results:
For now, I tested the pre-trained models shared by mmdetection using mmdetection tools. Also they provide a couple of tools to convert Right now I am trying to handle how can use TensorRT models in |
|
I deploy TensorRT engine to Python and I get some consistent results.
Warning In some parts of the video, you may see incorrect class names. For example, you might see both truck and car class names assigned to a vehicle. This is because I didn't run NMS when I deployed it in Python. I plan to fix this when I deploy it in C++. So, my next plan is doing same things in C++. Also, the detection times looks a little bit more. I did not understand the reason right now but I am working on it. |
xmfcx
changed the title
|
I am sharing the results from TensorRT deployment to C++, but currently, there are some issues with the results. I cannot see exactly the same results as with Python deployment. When I check the bounding box and score results, everything appears to be the same as in Python. However, when I check the labels, the results are very different. I am currently trying to resolve this issue.
You can find the scripts I used for deployment at these links: |
|
Fatih suggest to:
|
|
I just started to deploy RTMDet to ROS 2. Here is the repository: https://github.com/leo-drive/tensorrt_rtmdet/
I plan to complete the porting to ROS2 by performing the following steps:
|
|
✅ CUDA preprocessing is OK Preprocess Time Comparison per Image:
✅ NMS is added When I checked the model, I saw that the existing NMS algorithm within the model was already working. Despite NMS working, the reason we saw overlapping boxes was because the detected objects belonged to different classes. To prevent this, I added a simple NMS algorithm that selects the one with the higher score for overlapping detections.
NMS Time per Image
✅ Batch sizes which greater than
✅ Hard Coded Parts, Launch and Config files
🟨 Plugin Loading Autoware has the same logic that I used to load the TensorRT plugin. for (const auto & plugin_path : plugin_paths) {
int32_t flags{RTLD_LAZY};
void * handle = dlopen(plugin_path.c_str(), flags); // Plugin path is '/path/to/plugin.so'
if (!handle) {
logger_.log(nvinfer1::ILogger::Severity::kERROR, "Could not load plugin library");
}
}After compilation, a file with the extension '.so' is created. This file stored in Is there any information about can we handle this in Cmake. If we cannot, how can I provide the path to the file located inside the 'build' folder? I was able to load the plugin using the file paths below:
🟨 There are three precision option (
🟨 Post process and Visualization In current implementation visualization and postprocess parts works on CPU. I can't figure out how I can do these on the GPU.
Following table shows the total time for
❓Message Type for Outputs YOLOX semantic segmentation uses following message types:
There is no message defination for instance segmentation, so my plan is creating a new message type which combines the current detection message and new instance segmentation information. Should I create under autoware_msgs or tier4_autoware_msgs 📝 An RTX 3090 GPU was used for the time calculations and benchmarking |
|
When I ran 8 separate RTMDet nodes, I obtained the results in the table below, numbers represents the process times per image:
Computer Specifications
|
|
Following tables shows the process times for Computer Specifications
RTMDetSingle Camera
Multiple Camera
YOLOXSingle Camera
Multiple Camera
|
|
Discussion on new message type for instance segmentation results: https://github.com/orgs/autowarefoundation/discussions/5047 |
|
Last updates: https://www.youtube.com/watch?v=N8qrGAxzSJM |
|
Some options to use RTMDet results on camera-lidar pipeline: 1) Only run RTMDet and use roi outputs with current pipeline. Since the bounding box results from RTMDet is same with YOLOX, you can directly use RTMDet with current camera-lidar pipeline. 2) Fuse instance segmentation mask with clusters from euclidean clustering and assign clusters to label. The outputs of euclidean clustering are only point clouds, and it is It may perform better with objects that are close to each other and overlapping. I haven't tested it yet. code: https://github.com/StepTurtle/autoware.universe/tree/feat/mask_cluster_fusion 3) Fuse the point cloud with instance segmentation masks and take the points within the mask as object. code: https://github.com/StepTurtle/autoware.universe/tree/feat/mask_pointcloud_fusion 4) Fuse the point cloud with the instance segmentation mask. Filter out the points that do not correspond to objects within the mask and create a filtered point cloud. Couldn't find a useful place where we can use the filtered point cloud. PR: #8167 |
|
@StepTurtle Hi, first of all, thank you for your contribution to Autoware 🙏 However, I am not sure whether it would be better to merge this into Autoware at this moment (sorry for bringing this up after all of the reviews 🙏 ) I have several questions to ask:
Let me know your thoughts. |
It also has competitive speed. 
Why project boxes when you can project instance segmentation masks? ❌ Poor bounding box to point cloud performance
final-yolox.mp4ROI cluster fusion code has very complicated rules: ❌ Reduced performance with
|
I think this is a very backwards way of thinking. The universe is supposed to be for community contributions. A lot of effort is already been put into this integration. And the author was very cooperative during the process. The benefits are very clear and I don't see the reason for refusing to accept a feature like this. It doesn't even affect the existing repositories. I am very confused about your proposal. |
This is too late to suggest. All work has been done and it has to be merged to the main now. I am sorry for not giving any other options but these comments had to be made earlier. |
|
@xmfcx @armaganarsln @StepTurtle First, I would like to apologize if I have been disrespectful in any way. My primary concern was to understand "specifically which use cases posed challenges." However, now it make sense to me with the Fatih's comment here. Let me provide this feedback to our team too 🙏 |
|
@kminoda san, thank you and I am sorry for misunderstanding you on your approach. I thought you don't want it to be used or be in the main. Anyway there is still a lot work to be done for it to be useful so let's focus on that right now and see if there will be an improvement in the end in the overall false detections. |
|
Hey @kminoda , could you update me on your latest decision? Are the reviews ongoing, or are there other concerns you're currently considering? It would be helpful to know the latest status so I can continue working. |
Checklist
Description
We plan to add the RTMDet model in addition to the existing YOLOX model in Autoware Universe. While the YOLOX model is successful in the bounding box task, its instance segmentation layer is weak. We aim to improve instance segmentation results by adding RTMDet model.
Purpose
Our goal is to enhance the lidar image fusion pipeline by adding the RTMDet model to Autoware for image segmentation.
Possible approaches
We can convert pre-trained PyTorch models to ONNX and TensorRT formats, and we can create a ROS 2 package to handle the TensorRT models in Autoware Universe perception to implement them in Autoware.
Definition of done
The text was updated successfully, but these errors were encountered: