[c10d] add fake pg necessary collectives #102238
Conversation
This PR adds fake pg necessary collectives to enable e2e FSDP run with out multiprocess or multithreading [ghstack-poisoned]
wanchaol
requested review from
mrshenli,
zhaojuanmao,
rohan-varma,
H-Huang,
awgu,
kwen2501,
fegin,
kiukchung and
d4l3k
as code owners
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/102238
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit a21ea0a: This comment was automatically generated by Dr. CI and updates every 15 minutes. |
pytorch-bot
bot
added
the
release notes: distributed (c10d)
This PR adds fake pg necessary collectives to enable e2e FSDP run with out multiprocess or multithreading ghstack-source-id: 13e76fc541acab17e5a12a933f6ba033646bc06a Pull Request resolved: #102238
| # Fake process group doesn't need barrier | ||
| pass | ||
| elif backend == Backend.MPI: | ||
| if backend == Backend.MPI: |
There was a problem hiding this comment.
Nyehh, I think I preferred not mucking about with the environment variable instead haha.
There was a problem hiding this comment.
Yeah I kinda feel bad for this too, it was easier for me to leverage the previous work that uses a environ knob to control barrier init, let me think about it, there should be some better way for custom backend.
There was a problem hiding this comment.
landing this first to unblock dev efficiency and will address this in a follow up PR
There was a problem hiding this comment.
follow up PR here: #102343
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Stack from ghstack (oldest at bottom):
This PR adds fake pg necessary collectives to enable e2e FSDP run
with out multiprocess or multithreading