-
Notifications
You must be signed in to change notification settings - Fork 644
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(autoware_tensorrt_bevdet): add new 3d object detection method #7956
base: main
Are you sure you want to change the base?
feat(autoware_tensorrt_bevdet): add new 3d object detection method #7956
Conversation
Signed-off-by: liu cui <[email protected]>
Thank you for contributing to the Autoware project! 🚧 If your pull request is in progress, switch it to draft mode. Please ensure:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool PR 🚀
Let me provide some minor comments first. Also, please make sure that all the CIs are passing before opening the PR 🙏
@cyn-liu Please make sure that the node follows coding guildline: https://autowarefoundation.github.io/autoware-documentation/main/contributing/coding-guidelines/ros-nodes/class-design/ such as |
Signed-off-by: liu cui <[email protected]>
…e.universe into feat/add_tensorrt_bevdet Signed-off-by: liu cui <[email protected]>
Signed-off-by: liu cui <[email protected]>
…e.universe into feat/add_tensorrt_bevdet Signed-off-by: liu cui <[email protected]>
(Friendly ping 🙏 ) Would you mention us once all the CIs are fixed? |
I could see two required checks are not passed.
|
Signed-off-by: liu cui <[email protected]>
…e.universe into feat/add_tensorrt_bevdet Signed-off-by: liu cui <[email protected]>
@cyn-liu As this PR is stale for a week, from the maintenance perspective, let me make this PR a draft for now. Feel free to re-open the PR once all the CIs are fixed and ready to be reviewed. Thank you for your understanding, and we are looking forward to reviewing your PR. |
Signed-off-by: liu cui <[email protected]>
Signed-off-by: liu cui <[email protected]>
Signed-off-by: liu cui <[email protected]>
Signed-off-by: liu cui <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As a general comment, please follow the coding guideline in Autoware
https://autowarefoundation.github.io/autoware-documentation/main/contributing/coding-guidelines/languages/cpp/
perception/autoware_tensorrt_bevdet/include/autoware/tensorrt_bevdet/bevdet.hpp
Outdated
Show resolved
Hide resolved
perception/autoware_tensorrt_bevdet/include/autoware/tensorrt_bevdet/bevdet.hpp
Show resolved
Hide resolved
perception/autoware_tensorrt_bevdet/include/autoware/tensorrt_bevdet/bevdet.hpp
Outdated
Show resolved
Hide resolved
Signed-off-by: liu cui <[email protected]>
Signed-off-by: liu cui <[email protected]>
Signed-off-by: liu cui <[email protected]>
Signed-off-by: liu cui <[email protected]>
perception/autoware_tensorrt_bevdet/include/autoware/tensorrt_bevdet/bevdet.hpp
Outdated
Show resolved
Hide resolved
perception/autoware_tensorrt_bevdet/include/autoware/tensorrt_bevdet/bevdet.hpp
Outdated
Show resolved
Hide resolved
Signed-off-by: liu cui <[email protected]>
Signed-off-by: liu cui <[email protected]>
Signed-off-by: liu cui <[email protected]>
Signed-off-by: liu cui <[email protected]>
Signed-off-by: liu cui <[email protected]>
Signed-off-by: liu cui <[email protected]>
Signed-off-by: liu cui <[email protected]>
Signed-off-by: liu cui <[email protected]>
void BEVDet::mallocDeviceMemory() | ||
{ | ||
trt_buffer_sizes_.resize(trt_engine_->getNbBindings()); | ||
trt_buffer_dev_ = reinterpret_cast<void **>(new void *[trt_engine_->getNbBindings()]); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please apply the changes not only to the point we pointed out, but to all points related to the comment.
As we said in #7956 (comment), don't use new.
Please check our coding guidelines:
https://autowarefoundation.github.io/autoware-documentation/main/contributing/coding-guidelines/languages/cpp/
Our coding guideline follows CppCoreGuidelines
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Refactoring the entire open source project to fully meet the specifications is a significant workload, but I have to do so.
I found that the refactored code often encounters unknown problems during launch, and the cause of this problem has not been found. After restarting the node multiple times, it can be resolved.
Although the previous code did not meet your coding standards, it was simple clear, and can ran stably.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@cyn-liu
Is this what you meant in your comment?
autoware_tensorrt_bevdet/src/bevdet.cpp
was copied from another open source project.- The content written in the other open source project is being used as is.
If that's the case, instead of modifying the code, the following actions are also possible:
- Clearly indicate where the code was copied from, like in this example.
- Store all copied code in a
lib
directory.
For code outside the lib
, you have to follow Autoware's coding guidelines. For example, Autoware’s coding guidelines recommend avoiding the use of new
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@cyn-liu (cc. @Shin-kyoto)
With all due respect, but
Although the previous code did not meet your coding standards
All PR authors are asked to read and accept the coding standards in the contribution guidelines, so this is a requirement that was clear even before the work on this PR started, so we expect this to be followed. A large open source project cannot be managed otherwise and this is not a concept unique to Autoware.
it was simple clear
I disagree, and so do our coding standards and the C++ core guidelines. Also, this PR is almost 6000 lines which makes even perfectly simple and clear code hard to review thoroughly.
and can ran stably.
As a reviewer, this was impossible to verify due to all the non-standard and almost C-style C++. There is a reason that core guidelines and tools like Clang-Tidy are enforcing the use of smart pointers, disallowing reinterpret_cast etc.
If it does not run stably after the refactoring, that hints at race conditions and memory problems that still need to be addressed. Consider using AddressSanitizer and ThreadSanitizer to pinpoint the issues.
As this review is taking quite some hours to review for our employees, I would strongly suggest that the coding style issues are addressed, and that the PR is split up before occupying people with the same discussion over and over again.
If you do not want to comply with our coding standards, I am afraid we cannot merge this PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
friendly ping
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
friendly ping
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have talked with @Shin-kyoto on this.Like @mojomex mentioned, we have a coding standard in AWF which encourages the use of smart pointers instead of normal pointers. Therefore, we would like to comply to the guideline as much as possible.
However, I also understand the effort of refactoring the whole code, especially if the code was taken from another open source. If it is difficult to refactor, alternate option is to wrap the original BEVDet code as is as a vendor package and maintain it outside Autoware with different CIs settings and code standards. Then we can added through autoware.repos or through ROS Buildfarm and just call the libraries from Autoware Nodes. This might also make it easier for us to update the BEVDet implementation if any updates are made upstream.
Do you think the above approach is feasible? It might be difficult if you have already made a lot of modification to the BEVDet implementation so it would be nice if you can provide us how much modification you made from the original code.
I'm also okay to set up an online call if that makes you easier to discuss on this topic.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good Afternoon, Mits-san @mitsudome-r . thank you for the coordination.
- Second way is a better option for Continuously introduce great open source project into autoware
- I will tell Cynthia @cyn-liu to make a list of the modification point here
- it will be great if we can have a online call for this, especially for the topic “call the libraries from Autoware Nodes”, we are not familiar with this part
have a nice day!
Lucas
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The main modifications made to tensorrt-bevdet-cpp
library are as follows:
- Standardize log output by changing
print
,std:: cout
etc to ros2rclcpp logger
- Standardize the names of member functions and variables
- Modify multiple pointers to cpp
std::vector
- If the engine model does not exist, export engine before inference
Description
Integrating BEVDet into Autoware for 3D object detection based on multi-view images.
Related links
Issue Link
How was this PR tested?
This PR has been tested on local environment.
env1:
env2:
Notes for reviewers
Test using a bag containing Nuscenes data.
Note: The
frame_id
of/lidar_top
in this bag ismap
onnx model file
bag
Interface changes
Add a new
perception_mode
, If setperception_mode = camera
, the detector will launchtesnsorrt_bevdet
node.Topic changes
Additions and removals
~/input/topic_cloud
sensor_msgs::msg::PointCloud2
~/input/topic_img_fl
sensor_msgs::msg::Image
~/input/topic_img_f
sensor_msgs::msg::Image
~/input/topic_img_fr
sensor_msgs::msg::Image
~/input/topic_img_bl
sensor_msgs::msg::Image
~/input/topic_img_b
sensor_msgs::msg::Image
~/input/topic_img_br
sensor_msgs::msg::Image
~/output/painting_cloud
sensor_msgs::msg::PointCloud2
~/output/boxes
autoware_perception_msgs::msg::DetectedObjects
ROS Parameter Changes
Additions and removals
bev_detection_model
string
bevdet
Modifications
perception_mode
string
lidar
perception_mode
string
lidar
camera
Effects on system behavior
None.