[FEATURE]: Is it Possible to integrate Liger-Kernel?

[ericxsun]

Describe the feature

https://github.com/linkedin/Liger-Kernel

Liger Kernel is a collection of Triton kernels designed specifically for LLM training. It can effectively increase multi-GPU training throughput by 20% and reduce memory usage by 60%. We have implemented Hugging Face Compatible RMSNorm, RoPE, SwiGLU, CrossEntropy, FusedLinearCrossEntropy, and more to come.

[Edenzzzz]

Seems like a pretty light-weight library. cc @ver217 @isky-cd Any take on this? 😃

[isky-cd]

Liger Kernel is a collection of Triton kernels designed specifically for LLM training. It can effectively increase multi-GPU training throughput by 20% and reduce memory usage by 60%. We have implemented Hugging Face Compatible RMSNorm, RoPE, SwiGLU, CrossEntropy, FusedLinearCrossEntropy, and more to come.

I think this is a good attempt.

[ver217]

Does it compare with apex's implementation? We've integrate some apex cuda kernels and some of them are also implemented in Liger-kernel.

[Edenzzzz]

I think Apex only provides fused RMSNorm and LayerNorm kernels? They have some more