[SYCL] Implement Flash attention.

[qnixsynapse]

Currently Flash attention is available in CUDA and Metal backends in #5021.

From the paper: Flash attention is an IO-aware exact attention algorithm that uses tiling to reduce the number of memory reads/writes between GPU high bandwidth memory (HBM) and GPU on-chip SRAM. [...] it requires fewer HBM accesses than standard attention, and is optimal for a range of SRAM sizes. [..]

Thing is whether dedicated Intel GPUs can benefit from it or not and it will be interesting to see how much the performance improves.

[NeoZhangJianyu]

@qnixsynapse
Yes, Intel GPU will get benefit from it.
It has been verified in other AI framework.
We could consider to implement it in SYCL backend next.

[qnixsynapse]

@NeoZhangJianyu Nice. Thank you!

[qnixsynapse]

This issue has been tagged "stale" label. I am studying SYCL and C++ currently and waiting for major SYCL refactoring so that the code is readable and it will be easier for me to (eventually) implement flash attn kernel if needed.

Commented here to make this issue active again.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[piDack]

Any progress?