[Enhance] Show instance statistics before and after through pipeline #1863

JingweiZhang12 · 2022-09-23T09:20:24Z

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Show all statistics of the dataset: the length of the dataset and the number of instances per category in the dataset.
In order to debug in KITTI training, show the variance of the number of instances before and after through the pipeline by modifying the config. Here is an example:

train_dataloader = dict(
    ...,
    dataset=dict(
        type='KittiDataset',
        ...,
        show_ins_var=True))

The output information:

10/17 11:39:38 - mmengine - INFO - ------------------------------                                                                                                                                                                                             
10/17 11:39:38 - mmengine - INFO - The length of the dataset: 3712                                                                                                                                                                                            
10/17 11:39:38 - mmengine - INFO - The number of instances per category in the dataset:
+------------+--------+                                                                                                                                                                                                                                       
| category   | number |            
+------------+--------+                                                                                                                                                                                                                                       
| Pedestrian | 2207   |
| Cyclist    | 734    |                                                                                                        
| Car        | 22158  |
+------------+--------+                                                                                                                                        
10/17 11:43:40 - mmengine - INFO - The number of instances per category after and before through pipeline:
+------------+------------+------------+                                                                                                                                                                                                                      
| category   | new number | ori number |                                                                                                                                                                                                                      
+------------+------------+------------+                                                                                                                                                                                                                      
| Car        | 4          | 4          |                                                                                                                                                                                                                      
| Pedestrian | 12         | 15         |                                                                                                                                                                                                                      
| Cyclist    | 1          | 1          |                                                                                                                                                                                                                      
+------------+------------+------------+

Modification

Please briefly describe what modification is made in this PR.

BC-breaking (Optional)

Does the modification introduce changes that break the back-compatibility of the downstream repos?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.

Checklist

Pre-commit or other linting tools are used to fix the potential lint issues.
The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
If the modification has potential influence on downstream projects, this PR should be tested with downstream projects.
The documentation has been modified accordingly, like docstring or example tutorials.

Tai-Wang · 2022-09-28T01:28:42Z

Can this change show the instances of different categories? Better to support the category-wise statistics.

Tai-Wang · 2022-10-08T02:47:59Z

Can add an example print in the PR message?

mmdet3d/datasets/det3d_dataset.py

ZwwWayne · 2022-10-11T11:00:06Z

Suggest rebasing the code to the most recent dev.

mmdet3d/datasets/det3d_dataset.py

…stance_statistics

mmdet3d/datasets/det3d_dataset.py

…ection3d into instance_statistics

…into instance_statistics

…pen-mmlab#1863) * add instance statistics before and after through pipeline * add docstring * support showing cat-wise instance statistics * show all statistics of the dataset * small fix * polish code * show table * small fix * rename some varibles

* add mmengine assertion * add docstring * fix mminstall * fix semantic segmentation configs * fix semantic seg * fix lint * remove unused imports * fix * update pointnet2-s3dis config * update data_list according to scene_idxs * remove useless function * fix bug lack `eval_ann_info` during evaluation * fix bug * update doc * fix lint * fix dataset converter * update mmengine version * delete whitespace * [Fix] fix point cloud loop visualization error (#1914) * fix point cloud loop visualization error * fix browse_dataset * fix browse_dataset * support saving lidar_det Co-authored-by: JingweiZhang12 <[email protected]> * [Enhance] Show instance statistics before and after through pipeline (#1863) * add instance statistics before and after through pipeline * add docstring * support showing cat-wise instance statistics * show all statistics of the dataset * small fix * polish code * show table * small fix * rename some varibles * update task name and config for fcos3d++ on nus (#1927) * update docs * update docs * Update det3d_dataset.py * remove unnecessary optional docs * update docs * update docstrings * update docs * fix lint * update docs * [Refactor]: Refactor SASSD (#1901) * refactor sassd * rename some varibles * add ut for sassd * add some comments * add some comments * [Refactor] Use `mmeval.MeanIoU` for `SegMetric` (#1929) * Use mmeval.MeanIoU * fix comments and add self.reset * solve some comments * fix docstring Co-authored-by: ChaimZhu <[email protected]> * fix * fix * fix * fix lint * add args and kwargs * [Docs] Add mmyolo link in README (#1935) * [Refactor] Use `mmeval.MeanIoU` for `SegMetric` (#1929) * Use mmeval.MeanIoU * fix comments and add self.reset * add mmyolo Co-authored-by: ChaimZhu <[email protected]> * [Fix] fix seg metric issues and circle ci (#1962) * fix circle ci * update * update * [Enhance] Support evaluation for waymo dataset on s3 filesystem when tensorflow>=2.6.0 (#1958) * support tensorflow>2.6.0 * add filter conditions * [Fix]: change --pkl to --pkl-path in the data upgrade scripts (#1965) * [Refactor] update `metainfo` in pkl file and add `categories` into `metainfo` (#1934) * update metainfo in pkl file * update * add version for waymo * update kitti metric * [Refactor] Refactor FCAF3D (#1945) * add fcaf3d config * support inference of fcaf3d on scannet-3d * minor changes * fix config of scannet * align test precision of facaf3d && support training * add ut * fix bugs of dataset conversion and indoor_metric * support fcaf3d on sunrgbd datasets and add rorate_iou_loss ut * small fix * add docstring and typehint * add typehint for SparseTensor * remove pdb * fix fcaf3d ut * [Docs] Update Chinese documentation (#1891) * [Enhance] Add an example of pcd to bin format. (#1973) * [Enhance] Add an example of pcd to bin format. * Fix installation and typo. * [Fix]: fix image conversion of Waymo to avoid information loss (#1979) * add mmeval * [Refactor] rename `CLASSES` and `PALETTE` to `classes` and `palette` in dataset metainfo (#1932) * rame CLASS and PALETTE to class and palette * change mmcv-full to mmcv * fix comments * [CI]: fix CI (#2007) * Update docs && support loading old checkpoints in FCAF3D (#1974) * add loading converted keys and README, metafile * update readme * reorganize readme * add version in FCAF3d detector * fix converter error * [Fix] fix kitti evaluation bugs on test dataset (#2005) * fix kitti test evaluation bugs * update * [Fix] update the key name of modules from mmdet in cfgs (#2011) * [Fix] fix instance statistics when only detecting a single class (#2003) * Update iou_neg_piecewise_sampler.py * fix lint * [Enhance] Remove useless json file converter (#1971) * [Enhance] Refactor the import of `mmdet` (#1947) * fix config * refactor import of `mmdet` * fix lint * fix lint * [Enhance] Speed up evaluation on waymo (#2008) * support fast eval on waymo * support waymo evaluatioin more flexible and faster * support waymo evaluatioin more flexible and faster * renames * add docs * add guides for multi-thread evaluation toolkit * fix docstring * add download link for idx2metainfo * add docstring * set convert_kitti_format=False in Lidar-based methods * fix docs * add docstring * [Features] Support PV_RCNN modules (#1957) * add pvrcnn module code * add voxelsa * fix * fix comments * fix comments * fix comments * add stack sa * fix * fix comments * fix comments * fix * add ut * fix comments * [Fix]Fix a bug in StackQueryAndGroup (#2043) * fix a bug * fix a batch inference bug * fix docs * [Community] Add 'Projects/' folder, and the first example project (#2042) * add project in mmdet3d * fix * fix * [Fix] Fix waymo converter (#2040) * fix waymo converter * [Fix] Update waymo converter and fix lint * Update waymo_converter.py * Update kitti_converter.py * Update update_infos_to_v2.py * revert * fix paths in `update_infos_to_v2` on Windows by using `pathlib` (#2031) * fix paths on Windows by using `pathlib` * refactor Path import * simplify the warning and error when using open3d and MinkowskiEngine (#2027) * [FIX] replace DefaultFormatBundle/3D with Pack(3D)DetInputs (#1987) * replace defaultformatbundle3d with pack3ddetinputs * remove normalize, pad, imagetotensor from configs * rm unused key 'img_norm_cfg' * fix lint errors * fix lint errors * fix lint error * xx * [Docs] Refine the documentation (#1994) * refine doc * refine docs * replace `CLASSES` with `classes` * update doc * Minor fix Co-authored-by: Tai-Wang <[email protected]> * [Fix] Fix some loading bugs and support fov_image_based mode in Waymo dataset. (#1942) * modify sample_id to sample_id and support fov_image_based on waymo dataset * Update waymo_metric.py * Minor fix * Minor fix * Minor fix * Minor fix * Minor fix * Minor fix * Minor fixes * Minor fixes * Remove optional * fix dataset instances converting bugs * Add a blank line to fix the doc compilation format * Fix the bin file name in waymo_fov config * Resolve conflicts * fix ci and other things Co-authored-by: Tai-Wang <[email protected]> Co-authored-by: lianqing11 <[email protected]> Co-authored-by: ChaimZhu <[email protected]> * [Fix] Add num_features in Lyft dataset info file (#1948) Co-authored-by: Tai-Wang <[email protected]> * [Feature] Support FCAF3D on S3DIS dataset in `dev-1.x` branch (#1984) * support fcaf3d for s3dis dataset * Update convert_utils.py * Update seg3d_dataset.py * Delete compose.py * fix import error * use `mmengine.Compose` * Update s3dis-3d.py * Update fcaf3d_2xb8_s3dis-3d-5class.py * Update s3dis_dataset.py * update unittest for s3dis * update docs * use `mmcv.Compose` instead of `mmengine.Compose` * update docstring * fix s3dis preprocessing bug * Add typehint * Update config and fix s3dis dataset * update typehit * Update convert_utils.py * Update README and metafile Co-authored-by: Tai-Wang <[email protected]> * fix pp_waymo * update * resolve Co-authored-by: ChaimZhu <[email protected]> Co-authored-by: JingweiZhang12 <[email protected]> Co-authored-by: Qing Lian <[email protected]> Co-authored-by: Shaun <[email protected]> Co-authored-by: Wencheng Wu <[email protected]> Co-authored-by: Erli Ouyang <[email protected]> Co-authored-by: VVsssssk <[email protected]> Co-authored-by: vavanade <[email protected]> Co-authored-by: Tai-Wang <[email protected]> Co-authored-by: lianqing11 <[email protected]>

…2046) * add mmengine assertion * add docstring * fix mminstall * fix semantic segmentation configs * fix semantic seg * fix lint * remove unused imports * fix * update pointnet2-s3dis config * update data_list according to scene_idxs * remove useless function * fix bug lack `eval_ann_info` during evaluation * fix bug * update doc * fix lint * fix dataset converter * update mmengine version * delete whitespace * [Fix] fix point cloud loop visualization error (#1914) * fix point cloud loop visualization error * fix browse_dataset * fix browse_dataset * support saving lidar_det Co-authored-by: JingweiZhang12 <[email protected]> * [Enhance] Show instance statistics before and after through pipeline (#1863) * add instance statistics before and after through pipeline * add docstring * support showing cat-wise instance statistics * show all statistics of the dataset * small fix * polish code * show table * small fix * rename some varibles * update task name and config for fcos3d++ on nus (#1927) * update docs * update docs * Update det3d_dataset.py * remove unnecessary optional docs * update docs * update docstrings * update docs * fix lint * update docs * [Refactor]: Refactor SASSD (#1901) * refactor sassd * rename some varibles * add ut for sassd * add some comments * add some comments * [Refactor] Use `mmeval.MeanIoU` for `SegMetric` (#1929) * Use mmeval.MeanIoU * fix comments and add self.reset * solve some comments * fix docstring Co-authored-by: ChaimZhu <[email protected]> * fix * fix * fix * fix lint * add args and kwargs * [Docs] Add mmyolo link in README (#1935) * [Refactor] Use `mmeval.MeanIoU` for `SegMetric` (#1929) * Use mmeval.MeanIoU * fix comments and add self.reset * add mmyolo Co-authored-by: ChaimZhu <[email protected]> * [Fix] fix seg metric issues and circle ci (#1962) * fix circle ci * update * update * [Enhance] Support evaluation for waymo dataset on s3 filesystem when tensorflow>=2.6.0 (#1958) * support tensorflow>2.6.0 * add filter conditions * [Fix]: change --pkl to --pkl-path in the data upgrade scripts (#1965) * [Refactor] update `metainfo` in pkl file and add `categories` into `metainfo` (#1934) * update metainfo in pkl file * update * add version for waymo * update kitti metric * [Refactor] Refactor FCAF3D (#1945) * add fcaf3d config * support inference of fcaf3d on scannet-3d * minor changes * fix config of scannet * align test precision of facaf3d && support training * add ut * fix bugs of dataset conversion and indoor_metric * support fcaf3d on sunrgbd datasets and add rorate_iou_loss ut * small fix * add docstring and typehint * add typehint for SparseTensor * remove pdb * fix fcaf3d ut * [Docs] Update Chinese documentation (#1891) * [Enhance] Add an example of pcd to bin format. (#1973) * [Enhance] Add an example of pcd to bin format. * Fix installation and typo. * [Fix]: fix image conversion of Waymo to avoid information loss (#1979) * add mmeval * [Refactor] rename `CLASSES` and `PALETTE` to `classes` and `palette` in dataset metainfo (#1932) * rame CLASS and PALETTE to class and palette * change mmcv-full to mmcv * fix comments * [CI]: fix CI (#2007) * Update docs && support loading old checkpoints in FCAF3D (#1974) * add loading converted keys and README, metafile * update readme * reorganize readme * add version in FCAF3d detector * fix converter error * [Fix] fix kitti evaluation bugs on test dataset (#2005) * fix kitti test evaluation bugs * update * [Fix] update the key name of modules from mmdet in cfgs (#2011) * [Fix] fix instance statistics when only detecting a single class (#2003) * Update iou_neg_piecewise_sampler.py * fix lint * [Enhance] Remove useless json file converter (#1971) * [Enhance] Refactor the import of `mmdet` (#1947) * fix config * refactor import of `mmdet` * fix lint * fix lint * [Enhance] Speed up evaluation on waymo (#2008) * support fast eval on waymo * support waymo evaluatioin more flexible and faster * support waymo evaluatioin more flexible and faster * renames * add docs * add guides for multi-thread evaluation toolkit * fix docstring * add download link for idx2metainfo * add docstring * set convert_kitti_format=False in Lidar-based methods * fix docs * add docstring * [Features] Support PV_RCNN modules (#1957) * add pvrcnn module code * add voxelsa * fix * fix comments * fix comments * fix comments * add stack sa * fix * fix comments * fix comments * fix * add ut * fix comments * fix kitti mono_det bug * [Fix]Fix a bug in StackQueryAndGroup (#2043) * fix a bug * fix a batch inference bug * fix docs * [Community] Add 'Projects/' folder, and the first example project (#2042) * add project in mmdet3d * fix * fix * [Fix] Fix waymo converter (#2040) * fix waymo converter * [Fix] Update waymo converter and fix lint * Update waymo_converter.py * Update kitti_converter.py * Update update_infos_to_v2.py * revert * fix paths in `update_infos_to_v2` on Windows by using `pathlib` (#2031) * fix paths on Windows by using `pathlib` * refactor Path import * simplify the warning and error when using open3d and MinkowskiEngine (#2027) * [FIX] replace DefaultFormatBundle/3D with Pack(3D)DetInputs (#1987) * replace defaultformatbundle3d with pack3ddetinputs * remove normalize, pad, imagetotensor from configs * rm unused key 'img_norm_cfg' * fix lint errors * fix lint errors * fix lint error * xx * [Docs] Refine the documentation (#1994) * refine doc * refine docs * replace `CLASSES` with `classes` * update doc * Minor fix Co-authored-by: Tai-Wang <[email protected]> * Update converter * [Fix] Fix some loading bugs and support fov_image_based mode in Waymo dataset. (#1942) * modify sample_id to sample_id and support fov_image_based on waymo dataset * Update waymo_metric.py * Minor fix * Minor fix * Minor fix * Minor fix * Minor fix * Minor fix * Minor fixes * Minor fixes * Remove optional * fix dataset instances converting bugs * Add a blank line to fix the doc compilation format * Fix the bin file name in waymo_fov config * Resolve conflicts * fix ci and other things Co-authored-by: Tai-Wang <[email protected]> Co-authored-by: lianqing11 <[email protected]> Co-authored-by: ChaimZhu <[email protected]> * [Fix] Add num_features in Lyft dataset info file (#1948) Co-authored-by: Tai-Wang <[email protected]> * [Feature] Support FCAF3D on S3DIS dataset in `dev-1.x` branch (#1984) * support fcaf3d for s3dis dataset * Update convert_utils.py * Update seg3d_dataset.py * Delete compose.py * fix import error * use `mmengine.Compose` * Update s3dis-3d.py * Update fcaf3d_2xb8_s3dis-3d-5class.py * Update s3dis_dataset.py * update unittest for s3dis * update docs * use `mmcv.Compose` instead of `mmengine.Compose` * update docstring * fix s3dis preprocessing bug * Add typehint * Update config and fix s3dis dataset * update typehit * Update convert_utils.py * Update README and metafile Co-authored-by: Tai-Wang <[email protected]> * Update kitti_dataset.py * replace `List` wiith `Tuple` format to represent `origin` * Update convert_utils.py * add comments * update * resolve Co-authored-by: ChaimZhu <[email protected]> Co-authored-by: JingweiZhang12 <[email protected]> Co-authored-by: Qing Lian <[email protected]> Co-authored-by: Shaun <[email protected]> Co-authored-by: Wencheng Wu <[email protected]> Co-authored-by: Erli Ouyang <[email protected]> Co-authored-by: VVsssssk <[email protected]> Co-authored-by: vavanade <[email protected]> Co-authored-by: Tai-Wang <[email protected]> Co-authored-by: lianqing11 <[email protected]>

JingweiZhang12 changed the title ~~[Enhance] Show instance statistics before and after pipeline~~ [Enhance] Show instance statistics before and after through pipeline Sep 23, 2022

JingweiZhang12 requested review from VVsssssk and Tai-Wang September 29, 2022 11:58

Tai-Wang approved these changes Oct 8, 2022

View reviewed changes

ZCMax approved these changes Oct 9, 2022

View reviewed changes

ZwwWayne reviewed Oct 10, 2022

View reviewed changes

mmdet3d/datasets/det3d_dataset.py Outdated Show resolved Hide resolved

ZwwWayne reviewed Oct 10, 2022

View reviewed changes

mmdet3d/datasets/det3d_dataset.py Outdated Show resolved Hide resolved

VVsssssk reviewed Oct 10, 2022

View reviewed changes

mmdet3d/datasets/det3d_dataset.py Outdated Show resolved Hide resolved

JingweiZhang12 requested review from ZwwWayne and removed request for ZwwWayne October 10, 2022 14:11

VVsssssk approved these changes Oct 11, 2022

View reviewed changes

ZwwWayne reviewed Oct 11, 2022

View reviewed changes

mmdet3d/datasets/det3d_dataset.py Outdated Show resolved Hide resolved

ZwwWayne reviewed Oct 11, 2022

View reviewed changes

mmdet3d/datasets/det3d_dataset.py Show resolved Hide resolved

JingweiZhang12 added 6 commits October 11, 2022 21:09

add instance statistics before and after through pipeline

d352e3c

add docstring

0c5c4bd

support showing cat-wise instance statistics

8938bfa

show all statistics of the dataset

557ef80

small fix

9048654

polish code

2d308d8

JingweiZhang12 force-pushed the instance_statistics branch from 617716e to 2d308d8 Compare October 11, 2022 13:11

JingweiZhang12 added 2 commits October 11, 2022 21:28

show table

a4cedf8

small fix

99736a4

JingweiZhang12 requested review from ZwwWayne and jshilong and removed request for ZwwWayne and jshilong October 11, 2022 14:06

JingweiZhang12 requested review from ZwwWayne and removed request for ZwwWayne October 12, 2022 08:59

Merge branch 'dev-1.x' of github.com:open-mmlab/mmdetection3d into in…

f386a68

…stance_statistics

JingweiZhang12 requested review from ZwwWayne and removed request for ZwwWayne October 17, 2022 01:15

ZwwWayne reviewed Oct 17, 2022

View reviewed changes

mmdet3d/datasets/det3d_dataset.py Outdated Show resolved Hide resolved

mmdet3d/datasets/det3d_dataset.py Outdated Show resolved Hide resolved

mmdet3d/datasets/det3d_dataset.py Outdated Show resolved Hide resolved

mmdet3d/datasets/det3d_dataset.py Outdated Show resolved Hide resolved

JingweiZhang12 added 3 commits October 17, 2022 10:47

rename some varibles

8be94f6

Merge branch 'instance_statistics' of github.com:JingweiZhang12/mmdet…

69f1702

…ection3d into instance_statistics

Merge branch 'dev-1.x' of https://github.com/open-mmlab/mmdetection3d …

b778b69

…into instance_statistics

JingweiZhang12 requested a review from ZwwWayne October 17, 2022 03:36

ZwwWayne approved these changes Oct 17, 2022

View reviewed changes

ZwwWayne merged commit f0175bc into open-mmlab:dev-1.x Oct 17, 2022

JingweiZhang12 deleted the instance_statistics branch January 11, 2023 14:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhance] Show instance statistics before and after through pipeline #1863

[Enhance] Show instance statistics before and after through pipeline #1863

JingweiZhang12 commented Sep 23, 2022 •

edited

Loading

Tai-Wang commented Sep 28, 2022

Tai-Wang commented Oct 8, 2022

ZwwWayne commented Oct 11, 2022

[Enhance] Show instance statistics before and after through pipeline #1863

[Enhance] Show instance statistics before and after through pipeline #1863

Conversation

JingweiZhang12 commented Sep 23, 2022 • edited Loading

Motivation

Modification

BC-breaking (Optional)

Use cases (Optional)

Checklist

Tai-Wang commented Sep 28, 2022

Tai-Wang commented Oct 8, 2022

ZwwWayne commented Oct 11, 2022

JingweiZhang12 commented Sep 23, 2022 •

edited

Loading