[PyTorch] Should I use DistributedDataParallel ?

[RobinFrcd]

Hello,
I'm currently taking a look at hyperparameter_tuning/pytorch_mnist/mnist.py. In this example, there's the use of torch.nn.parallel.DistributedDataParallel(model).

According to the Sagemaker Doc, "When using data parallelism with DDP, the DistributedDataParallel wrapper is not supported."

So what's the truth/best practice here ?

Thanks

[mchoi8739]

Hi,

The example you are looking at is using the native PyTorchs DistributedDataParallel modules.

The SageMaker doc you looked at is about the SageMaker's model parallel library. The doc is saying that SageMaker's model parallel library does not support the native PyTorch's DistributedDataParallel modules. Instead, you can use SageMaker's data parallel library that's compatible with the model parallel library.

The SageMaker data parallel library provides DDP modules equivalent to torch.nn.parallel.DistributedDataParallel. The SageMaker's distributed data parallelism modules and how to adapt your PyTorch training scripts are documented in https://docs.aws.amazon.com/sagemaker/latest/dg/data-parallel-modify-sdp-pt.html.
Basically, you need to replace the native PT DistributedDataParallel to from smdistributed.dataparallel.torch.parallel.distributed import DistributedDataParallel as DDP, and make a few more modifications to wrap modules of your model (optimizer, model, etc).

The two SageMaker distributed training libraries are benchmark tested and can help improve your model training on AWS infrastructure.

Are you looking for an example of running a PyTorch training job with model parallelism?

[RobinFrcd]

Thank you for your answer !

To be honest, my model is pretty small, but I have a lot of data. I'm looking for an example to run a PyTorch training with data-parallelism that could run both on a local machine with 2 GPUs and on Sagemaker.

Thanks!

[mchoi8739]

I see. Then I would recommend you to have a look at these docs and examples of using SageMaker's data parallelism.

• https://docs.aws.amazon.com/sagemaker/latest/dg/data-parallel.html
• https://docs.aws.amazon.com/sagemaker/latest/dg/distributed-training-notebook-examples.html

Let me know if you have further questions!

[mchoi8739]

ps. the SageMaker distributed data parallel library is optimized to run SageMaker training jobs (TF, PT, HuggingFace) on EC2 instances (EFA enabled) with 8 GPUs.

[RobinFrcd]

Thank you again for your answer, I'll go through all the doc then !