Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

any plans to add improved PP in llama-3? #483

Closed
SeunghyunSEO opened this issue Jul 25, 2024 · 3 comments
Closed

any plans to add improved PP in llama-3? #483

SeunghyunSEO opened this issue Jul 25, 2024 · 3 comments

Comments

@SeunghyunSEO
Copy link

batch constraint removed Pipeline Parallelism (PP) in llama-3 paper seems to added torch 2.4.
do you have any plans to add this feature to torchtitan ?!
@tianyu-l
Copy link
Contributor

@wconstab @H-Huang anything to share?

@H-Huang
Copy link
Member

H-Huang commented Jul 26, 2024

Hi @SeunghyunSEO, the ScheduleFlexibleInterleaved1F1B was not introduced in 2.4, but it is available in the PyTorch nightly build (and will be available in the 2.5 release).

For torchtitan I can quickly add a PR to add a flag to support this schedule.

@H-Huang H-Huang mentioned this issue Jul 29, 2024
Merged
H-Huang added a commit that referenced this issue Jul 29, 2024
@H-Huang
Update base for Update on "[PP] add flexible interleaved 1f1b schedule"
3840add 
fixes #483


[ghstack-poisoned]
H-Huang added a commit that referenced this issue Jul 29, 2024
@H-Huang
Update on "[PP] add flexible interleaved 1f1b schedule"
3b2c865 
fixes #483


[ghstack-poisoned]
H-Huang added a commit that referenced this issue Jul 30, 2024
@H-Huang
Update base for Update on "[PP] add flexible interleaved 1f1b schedule"
40c6398 
fixes #483

`python test_runner.py ./out --test pp_looped_flexible_1f1b`


[ghstack-poisoned]
H-Huang added a commit that referenced this issue Jul 30, 2024
@H-Huang
Update on "[PP] add flexible interleaved 1f1b schedule"
63dc3df 
fixes #483

`python test_runner.py ./out --test pp_looped_flexible_1f1b`


[ghstack-poisoned]
H-Huang added a commit that referenced this issue Jul 30, 2024
@H-Huang
[PP] add flexible interleaved 1f1b schedule (#490)
437550e 
Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at
bottom):
* __->__ #490

fixes #483

`python test_runner.py ./out --test pp_looped_flexible_1f1b`
@H-Huang
Copy link
Member

H-Huang commented Jul 30, 2024

merged in 9cf4b2f

@H-Huang H-Huang closed this as completed Jul 30, 2024
tianyu-l pushed a commit that referenced this issue Aug 16, 2024
@H-Huang
Update on "[PP] add flexible interleaved 1f1b schedule"
84e40b2 
fixes #483


[ghstack-poisoned]
tianyu-l pushed a commit that referenced this issue Aug 16, 2024
@H-Huang
Update on "[PP] add flexible interleaved 1f1b schedule"
929e6e2 
fixes #483

`python test_runner.py ./out --test pp_looped_flexible_1f1b`


[ghstack-poisoned]
tianyu-l pushed a commit that referenced this issue Aug 16, 2024
@H-Huang
[PP] add flexible interleaved 1f1b schedule (#490)
0322d27 
Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at
bottom):
* __->__ #490

fixes #483

`python test_runner.py ./out --test pp_looped_flexible_1f1b`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@H-Huang @SeunghyunSEO @tianyu-l