[Train] Change default NCCL_SOCKET_IFNAME to blacklist veth
#31824
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: amogkam <[email protected]>
Signed-off-by: amogkam <[email protected]>
Signed-off-by: amogkam <[email protected]>
|
Passing distributed GPU training test on |
Signed-off-by: amogkam <[email protected]>
|
nice! From the logs, looks like NCCL is still seeing veth in some of the nodes. Any idea why? I see 12 worker logs with |
cadedaniel
approved these changes
Jan 21, 2023
There was a problem hiding this comment.
discussed offline. we're not sure if that line is printed before or after blacklist. we should catch it in nightly if the order is nondeterministic.
can we file an issue against product to remove the virtual ethernet interface? then Ray Train can get out of the way of users completely here (default to NCCL behavior).
cadedaniel
requested changes
Jan 21, 2023
python/ray/train/constants.py
Outdated
| # "en". | ||
| DEFAULT_NCCL_SOCKET_IFNAME = "en,eth,bond" | ||
| # Blacklist virtualized networking. | ||
| DEFAULT_NCCL_SOCKET_IFNAME = "^lo,docker,vethc" |
There was a problem hiding this comment.
oh actually the reason is that it should be veth not vethc. c happened to be the first hex character of the ID I sent you.
that also explains why it was present in only 12 or the 16 -- the other 4 must have had c as the first character of the id.
There was a problem hiding this comment.
Ah good catch...then now I'm wondering why this passed. Maybe this is not needed at all 🤔. Let me run the test with this default removed.
Signed-off-by: amogkam <[email protected]>
Signed-off-by: amogkam <[email protected]>
cadedaniel
approved these changes
Jan 23, 2023
krfricke
reviewed
Jan 23, 2023
amogkam
changed the title
NCCL_SOCKET_IFNAME to blacklist vethcNCCL_SOCKET_IFNAME to blacklist veth
|
Updated the title-- it should be |
8 tasks
jjyao
pushed a commit
that referenced
this pull request
Jan 8, 2024
When running distributed Pytorch without GPUs, Pytorch selects a localhost interface for gloo (i.e. 127.0.0.1:XXX), breaking distributed training. This method in Pytorch can yield the incorrect interface when a) the the hostname resolves locally to the loopback address or b) when hostname lookups fail. This is scoped to DBR specifically because eth0 is guaranteed to exist there. Pytorch+Gloo does not support deny-listing like NCCL (as we do in #31824) because Pytorch directly uses the environment variable GLOO_SOCKET_IFNAME as the interface to use https://github.com/pytorch/pytorch/blob/7956ca16e649d86cbf11b6e122090fa05678fac3/torch/csrc/distributed/c10d/init.cpp#L2243. Signed-off-by: Ian Rodney <[email protected]>
vickytsang
pushed a commit
to ROCm/ray
that referenced
this pull request
Jan 12, 2024
…2202) When running distributed Pytorch without GPUs, Pytorch selects a localhost interface for gloo (i.e. 127.0.0.1:XXX), breaking distributed training. This method in Pytorch can yield the incorrect interface when a) the the hostname resolves locally to the loopback address or b) when hostname lookups fail. This is scoped to DBR specifically because eth0 is guaranteed to exist there. Pytorch+Gloo does not support deny-listing like NCCL (as we do in ray-project#31824) because Pytorch directly uses the environment variable GLOO_SOCKET_IFNAME as the interface to use https://github.com/pytorch/pytorch/blob/7956ca16e649d86cbf11b6e122090fa05678fac3/torch/csrc/distributed/c10d/init.cpp#L2243. Signed-off-by: Ian Rodney <[email protected]>
8 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: amogkam [email protected]
Closes #30333.
Previously, we would set a default NCCL interface whitelist in Ray Train to prioritize ethernet. This is to avoid this issue: https://github.com/anyscale/product/issues/8310.
However, this default whitelist is not fully exhaustive, and prevents users from doing distributed GPU training over wireless: #30333.
Instead, we change to a blacklist so that NCCL does not use
vethinterface which resolves both issues (thanks @cadedaniel for identifying this!)Why are these changes needed?
Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.