CombinedStreamingDataset causes NCCL timeout when using multiple nodes #340
Comments
|
Hi! thanks for your contribution!, great first issue! |
|
Hey @hubenjm, Did you provide Could you share a reproducible script or the code of your training dataset ? |
|
Hey @hubenjm. Any updates ? |
|
@tchaton Thanks for the suggestions. I am currently trying to run my code again while explicitly setting |
|
OK, that did not work either, so I am going to have to work on creating a simpler example code to share that replicates the problem |
|
To replicate the problem, you first need to run Then after that data is generated in s3, to submit a training job in SageMaker with e.g. 2 nodes, follow the If you want to run the training code in your own cluster via or replace NOTE that I only replicated the error using the SageMaker training job approach above, but I don't think there's any significant difference between running it there versus on a self-managed cluster, since under the hood Sagemaker will execute a very similar torchrun command as above. With above code example and arguments used I got a softlock to occur at around epoch 5 with 2 nodes. With 1 node it runs fine. |
|
OK, another update. |
|
My next step will be to try getting a multi node sagemaker training job working with the same code but replacing the dataset/data loader with standard torch dataset and DataLoader class. If that doesn't work then I suppose this issue is moot and the problem is something else. But it would be very useful to a lot of folks in general to be able to use LitData and Lightning effectively with multinode sagemaker training jobs. |
|
Hey @hubenjm Could you check the dataset length or the number of batch read on each rank ? This can happen if somehow the length wasn't inferred properly and one rank gets more data. |
and others
🐛 Bug
I'm running a training job with 2 nodes in SageMaker using torchrun to launch. I'm using a CombinedStreamingDataset for the training dataset and using
train_weight_factors = [0.8,0.07,0.07,0.07]. The training stops printing out log messages after some fixed number of batches (depending on random seed I guess). Where the training stops is deterministic if seed is fixed, based on my experiments. Then the NCCL timeout triggers an exception after 30 minutes. The training code works fine on a single node though.To Reproduce
Use CombinedStreamingDataset for training dataset with
train_weight_factorsnotNoneanditerate_over_all = False. Launch training withtorchrunwith num_nodes > 1.Code sample
Expected behavior
Training should not softlock in the middle of an epoch
Environment
conda,pip, source): SageMaker prebuilt deep learning container (763104351884.dkr.ecr.us-west-2.amazonaws.com/pytorch-training:2.3.0-gpu-py311-cu121-ubuntu20.04-sagemaker, see https://github.com/aws/deep-learning-containers/blob/master/available_images.md)Additional context
If you have any other suggestions about why multi-node training with CombinedDataset would fail like this, any help is appreciated.
The text was updated successfully, but these errors were encountered: