-
Notifications
You must be signed in to change notification settings - Fork 443
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add custom max iou assigner to prevent CPU OOM in training phase #2228
Conversation
Co-authored-by: Sungman Cho <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the nice work. BTW, how about using our own docstring instead of the original one for CustomMaxIoUAssigner
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the great work. Minor comment and question from my side:
otx/algorithms/detection/adapters/mmdet/models/assigners/custom_max_iou_assigner.py
Outdated
Show resolved
Hide resolved
…m_max_iou_assigner.py Co-authored-by: Songki Choi <[email protected]>
Summary
When a training image contain too many gt(over > 10000), current assigner, matching anchor bbox and gt, raises CPU OOM.
This is because assigner makes too large cpu tensor matrix. To fix this problem, this PR add a new custom max iou assigner.
This new custom max iou assigner splits gt bboxes into reasonable size when gt bboxes is too large. This may lead time increase but prevent sudden cpu oom.
How to test
In my desktop which has 64GB RAM cannot train Kiemgetal dataset. However after this fix, my desktop can train Kiemgetal dataset.
Checklist
License
Feel free to contact the maintainers if that's a concern.