Scan vs. Flash Attention profiling on Figure 8 (a)

[llmexperiment]

Hi All,

I am trying to re-produce the Figure 8 (left). FlashAttention vs. Scan.

I know this is already talked on the forum, but I am having similar issue where the scan is very slow.
@tridao , @albertfgu , @MzeroMiko, @apoorv2904, -- could you guys point me the code that is used to benchmark the scan?

This forum talked about this, but I could not reproduce the similar result (my scan is very slow in my GPU).

• How to compute MACs or FLOPs of mamba #110