Skip to content

linbinskn/context_parallel

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
benchmark
benchmark
 
 
dist_attn
dist_attn
 
 
ring_flash_attn
ring_flash_attn
 
 
test
test
 
 
test_attention
test_attention
 
 
README.md
README.md
 
 
setup.py
setup.py
 
 

Repository files navigation

Ulysses/Ring/FastSeq Attention

This repo implements the Ulysses, Ring and FastSeq Attention.

Test

PYTHONPATH=. torchrun --nproc_per_node 8 test_attention/test_attention_ulysses.py
PYTHONPATH=. torchrun --nproc_per_node 8 test_attention/test_attention_ring_zigzag.py
PYTHONPATH=. torchrun --nproc_per_node 8 test_attention/test_attention_fastseq.py

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

1 star

Watchers

4 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages