-
Notifications
You must be signed in to change notification settings - Fork 110
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
flash attention support #17
Comments
FlashAttention and related would be someting that is useful on GPU (only/mostly). It could get a 2-4x speedup for Bert. By best guess would be to add Contributions are of course also welcome. |
I'll add FlashAttention via BetterTransformers: |
infinity uses now Beyond, you can set the torch backend to infinity/libs/infinity_emb/infinity_emb/transformer/sentence_transformer.py Lines 53 to 60 in 8116680
Perhaps, the integration better with upcoming versions of |
Relevant issues & prs
huggingface/transformers#26557
huggingface/transformers#26350
huggingface/transformers#26585
The text was updated successfully, but these errors were encountered: