Does TP/SP overlap support nccl backend?

[wplf]

Hi, thank you for great works.

I've seen the source code and found that TE linear TP/SP overlap supports reduce_scatter and all_gather in nccl backend.
In the megatron repo, TP overlap is initialized on mpi. But, torch don't support mpi backend by default.

So, does TP/SP overlap support nccl backend?

[yaox12]

Yes, we now support bootstrap from NCCL.
And in latest Megatron, you can find the argument --tp-comm-bootstrap-backend in https://github.com/NVIDIA/Megatron-LM/blob/d357c188323b6928cbcbd6f7e06af04c1694382f/megatron/training/arguments.py#L1163-L1165

[wplf]

Thank you, very much.
I'll give it a try.