Questions about computing FLOPs with fvcore

[exhyy]

Hi all, thanks for your great work!
I am wandering how to compute the FLOPs reported in https://github.com/OpenGVLab/VideoMamba/blob/main/videomamba/image_sm/MODEL_ZOO.md. Take videomamba_tiny as example, I have tried running python videomamba/image_sm/models/videomamba.py with changing the model to videomamba_tiny. Then I got the computed FLOPs, 1.08G, which equals to the reported one. However, if I use the following piece of code:

model = videomamba_tiny().cuda()
flops = FlopCountAnalysis(model, torch.rand(1, 3, img_size, img_size).cuda())
print(flops.by_operator())

I will get some warnings:

Unsupported operator aten::add encountered 72 time(s)
Unsupported operator aten::mul encountered 95 time(s)
Unsupported operator aten::mul_ encountered 24 time(s)
Unsupported operator aten::exp encountered 48 time(s)
Unsupported operator aten::neg encountered 48 time(s)
Unsupported operator prim::PythonOp.MambaInnerFnNoOutProj encountered 48 time(s)
Unsupported operator aten::flip encountered 48 time(s)
Unsupported operator aten::rand encountered 23 time(s)
Unsupported operator aten::floor_ encountered 23 time(s)
Unsupported operator aten::div encountered 23 time(s)
The following submodules of the model were never called during the trace of the graph. They may be unused, or they were accessed by direct calls to .forward() or via other python methods. In the latter case they will have zeros for statistics, though their statistics will still contribute to their parent calling module.
layers.0.mixer.act, layers.0.mixer.conv1d, layers.0.mixer.conv1d_b, layers.0.mixer.dt_proj, layers.0.mixer.dt_proj_b, layers.0.mixer.in_proj, layers.0.mixer.out_proj, layers.0.mixer.x_proj, layers.0.mixer.x_proj_b, layers.1.mixer.act, layers.1.mixer.conv1d, layers.1.mixer.conv1d_b, layers.1.mixer.dt_proj, layers.1.mixer.dt_proj_b, layers.1.mixer.in_proj, layers.1.mixer.out_proj, layers.1.mixer.x_proj, layers.1.mixer.x_proj_b, layers.10.mixer.act, layers.10.mixer.conv1d, layers.10.mixer.conv1d_b, layers.10.mixer.dt_proj, layers.10.mixer.dt_proj_b, layers.10.mixer.in_proj, layers.10.mixer.out_proj, layers.10.mixer.x_proj, layers.10.mixer.x_proj_b, layers.11.mixer.act, layers.11.mixer.conv1d, layers.11.mixer.conv1d_b, layers.11.mixer.dt_proj, layers.11.mixer.dt_proj_b, layers.11.mixer.in_proj, layers.11.mixer.out_proj, layers.11.mixer.x_proj, layers.11.mixer.x_proj_b, layers.12.mixer.act, layers.12.mixer.conv1d, layers.12.mixer.conv1d_b, layers.12.mixer.dt_proj, layers.12.mixer.dt_proj_b, layers.12.mixer.in_proj, layers.12.mixer.out_proj, layers.12.mixer.x_proj, layers.12.mixer.x_proj_b, layers.13.mixer.act, layers.13.mixer.conv1d, layers.13.mixer.conv1d_b, layers.13.mixer.dt_proj, layers.13.mixer.dt_proj_b, layers.13.mixer.in_proj, layers.13.mixer.out_proj, layers.13.mixer.x_proj, layers.13.mixer.x_proj_b, layers.14.mixer.act, layers.14.mixer.conv1d, layers.14.mixer.conv1d_b, layers.14.mixer.dt_proj, layers.14.mixer.dt_proj_b, layers.14.mixer.in_proj, layers.14.mixer.out_proj, layers.14.mixer.x_proj, layers.14.mixer.x_proj_b, layers.15.mixer.act, layers.15.mixer.conv1d, layers.15.mixer.conv1d_b, layers.15.mixer.dt_proj, layers.15.mixer.dt_proj_b, layers.15.mixer.in_proj, layers.15.mixer.out_proj, layers.15.mixer.x_proj, layers.15.mixer.x_proj_b, layers.16.mixer.act, layers.16.mixer.conv1d, layers.16.mixer.conv1d_b, layers.16.mixer.dt_proj, layers.16.mixer.dt_proj_b, layers.16.mixer.in_proj, layers.16.mixer.out_proj, layers.16.mixer.x_proj, layers.16.mixer.x_proj_b, layers.17.mixer.act, layers.17.mixer.conv1d, layers.17.mixer.conv1d_b, layers.17.mixer.dt_proj, layers.17.mixer.dt_proj_b, layers.17.mixer.in_proj, layers.17.mixer.out_proj, layers.17.mixer.x_proj, layers.17.mixer.x_proj_b, layers.18.mixer.act, layers.18.mixer.conv1d, layers.18.mixer.conv1d_b, layers.18.mixer.dt_proj, layers.18.mixer.dt_proj_b, layers.18.mixer.in_proj, layers.18.mixer.out_proj, layers.18.mixer.x_proj, layers.18.mixer.x_proj_b, layers.19.mixer.act, layers.19.mixer.conv1d, layers.19.mixer.conv1d_b, layers.19.mixer.dt_proj, layers.19.mixer.dt_proj_b, layers.19.mixer.in_proj, layers.19.mixer.out_proj, layers.19.mixer.x_proj, layers.19.mixer.x_proj_b, layers.2.mixer.act, layers.2.mixer.conv1d, layers.2.mixer.conv1d_b, layers.2.mixer.dt_proj, layers.2.mixer.dt_proj_b, layers.2.mixer.in_proj, layers.2.mixer.out_proj, layers.2.mixer.x_proj, layers.2.mixer.x_proj_b, layers.20.mixer.act, layers.20.mixer.conv1d, layers.20.mixer.conv1d_b, layers.20.mixer.dt_proj, layers.20.mixer.dt_proj_b, layers.20.mixer.in_proj, layers.20.mixer.out_proj, layers.20.mixer.x_proj, layers.20.mixer.x_proj_b, layers.21.mixer.act, layers.21.mixer.conv1d, layers.21.mixer.conv1d_b, layers.21.mixer.dt_proj, layers.21.mixer.dt_proj_b, layers.21.mixer.in_proj, layers.21.mixer.out_proj, layers.21.mixer.x_proj, layers.21.mixer.x_proj_b, layers.22.mixer.act, layers.22.mixer.conv1d, layers.22.mixer.conv1d_b, layers.22.mixer.dt_proj, layers.22.mixer.dt_proj_b, layers.22.mixer.in_proj, layers.22.mixer.out_proj, layers.22.mixer.x_proj, layers.22.mixer.x_proj_b, layers.23.mixer.act, layers.23.mixer.conv1d, layers.23.mixer.conv1d_b, layers.23.mixer.dt_proj, layers.23.mixer.dt_proj_b, layers.23.mixer.in_proj, layers.23.mixer.out_proj, layers.23.mixer.x_proj, layers.23.mixer.x_proj_b, layers.3.mixer.act, layers.3.mixer.conv1d, layers.3.mixer.conv1d_b, layers.3.mixer.dt_proj, layers.3.mixer.dt_proj_b, layers.3.mixer.in_proj, layers.3.mixer.out_proj, layers.3.mixer.x_proj, layers.3.mixer.x_proj_b, layers.4.mixer.act, layers.4.mixer.conv1d, layers.4.mixer.conv1d_b, layers.4.mixer.dt_proj, layers.4.mixer.dt_proj_b, layers.4.mixer.in_proj, layers.4.mixer.out_proj, layers.4.mixer.x_proj, layers.4.mixer.x_proj_b, layers.5.mixer.act, layers.5.mixer.conv1d, layers.5.mixer.conv1d_b, layers.5.mixer.dt_proj, layers.5.mixer.dt_proj_b, layers.5.mixer.in_proj, layers.5.mixer.out_proj, layers.5.mixer.x_proj, layers.5.mixer.x_proj_b, layers.6.mixer.act, layers.6.mixer.conv1d, layers.6.mixer.conv1d_b, layers.6.mixer.dt_proj, layers.6.mixer.dt_proj_b, layers.6.mixer.in_proj, layers.6.mixer.out_proj, layers.6.mixer.x_proj, layers.6.mixer.x_proj_b, layers.7.mixer.act, layers.7.mixer.conv1d, layers.7.mixer.conv1d_b, layers.7.mixer.dt_proj, layers.7.mixer.dt_proj_b, layers.7.mixer.in_proj, layers.7.mixer.out_proj, layers.7.mixer.x_proj, layers.7.mixer.x_proj_b, layers.8.mixer.act, layers.8.mixer.conv1d, layers.8.mixer.conv1d_b, layers.8.mixer.dt_proj, layers.8.mixer.dt_proj_b, layers.8.mixer.in_proj, layers.8.mixer.out_proj, layers.8.mixer.x_proj, layers.8.mixer.x_proj_b, layers.9.mixer.act, layers.9.mixer.conv1d, layers.9.mixer.conv1d_b, layers.9.mixer.dt_proj, layers.9.mixer.dt_proj_b, layers.9.mixer.in_proj, layers.9.mixer.out_proj, layers.9.mixer.x_proj, layers.9.mixer.x_proj_b
1079579328
Counter({'matmul': 697171968, 'linear': 348777984, 'conv': 28901376, 'layer_norm': 4728000})

I don't understand how fvcore computes FLOPs, but these warnings and the result of flops.by_operator() seem to indicate that this calculation method is not correct.

[Andy1621]

Good question! Since I'm also unfamiliar with how fvcore runs, maybe you can check how VMamba calculates GFLOPs manually.

[exhyy]

Good question! Since I'm also unfamiliar with how fvcore runs, maybe you can check how VMamba calculates GFLOPs manually.

Thanks for your replay. I just found the answer at state-spaces/mamba#110, which is used by VMamba.