Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Performance] The GEMM performance with the column major B matrix is not as good as row major B matrix. #2354

Open
chengjunlu opened this issue Sep 26, 2024 · 4 comments
Assignees
Labels
enhancement New feature or request performance

Comments

@chengjunlu
Copy link
Contributor

The performance gap is found in #2347

Need to investigate root cause of the performance drops of the column major B matrix case.
Roughly 1.5x worse than the row major B matrix case.

(I): Detected 7680 spills, recompiling the kernel using large GRF mode
(I): Kernel has now 0 spills
✅ Triton and Torch match
Time for torch: 0.31633758544921875 ms
Time for triton: 0.44517597556114197 ms
Compute A x B.T
OpenCL API not available for this operation
OpenCL API not available for this operation
OpenCL API not available for this operation
OpenCL API not available for this operation
(I): Detected 7680 spills, recompiling the kernel using large GRF mode
(I): Kernel has now 0 spills
✅ Triton and Torch match
Time for torch: 0.3375360071659088 ms
Time for triton: 0.6348815560340881 ms

@Egor-Krivov
Copy link
Contributor

I think this issue is essential for GEMM perf. Very often weights are stored with K dimensions as the last. Even pytorch linear layer does that: weight torch.Tensor – the learnable weights of the module of shape : (out_features, in_features)

https://pytorch.org/docs/stable/generated/torch.nn.Linear.html

@alexbaden
Copy link
Contributor

Adding to this, if the A matrix is column-major we have similar problems.

@Egor-Krivov
Copy link
Contributor

Egor-Krivov commented Oct 11, 2024

We now have microbenchmarks to track this performance. Currently GeoMean for onednn is ~90-100TFLOPs for both cases of A.T@B and for [email protected].

[email protected] for triton currently stands at ~60TFLOPs. Dashboard gemm-bt
A.T@B for triton currently stands at ~30TFLOPs, it significantly improved and was ~15TFLOPs recently. Dashboard gemm-at

So onednn is 1.5 times faster for B.T and 3 times faster for A.T

@Egor-Krivov
Copy link
Contributor

@alexbaden Should we change the title to reflect issue with A.T as well or create separate issue for that case?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request performance
Projects
None yet
Development

No branches or pull requests

4 participants