Fluid distributed training TODO

[Yancey1989]

Fluid Distribute Training Features

• code cleanup and polish
• implement LARS to improve training performance, Improve distributed training with LARS #6811
• fault-tolerant
  • checkpointing and recovering parameters on pserver
  • recover reader offset(may need master and etcd)
  • trainer pre-fetch parameters from pserver after the restart
• async training, fluid support asynchronous training #9941
• distributed data reader(should unify with single machine reader)
• calculate global AUC with the distributed table
• initialize trainable parameters from saved parameters on a trainer
• ring-base architecture to improve training performance
• distributed lookup table, https://github.com/PaddlePaddle/Paddle/projects/56
• full overlapping with parallel-executor on dist training
• split send_op into multiple send_vars_op and fetch_vars_op, Split send_op into fetch_vars_op and send_vars_op #9161

EDL

• implement the master process to schedule task
• etcd operator
• implement CRD to support kubernetes v1.8

Support different communication library

• gRPC performance enhancement
• OpenMPI with RDMA and GPU direct
• NCCL2 with multiple nodes
• follow up bRPC

Experiment

• different distributed training strategy (sync, async etc...) influence on model accuracy/throughput

CE

• Auto execute benchmark-job on AWS and generate a report

Future

• differences between multi-machine-single-device and multi-machine-multi-device
• better integration with single-machine training
• think about more flexible user-customized device placement for multi-machine training.
• need to discuss whether we need the remote executor

[panyx0718]

Some extra that might worth adding:

distributed data reader (should unify with single machine reader)
evaluate different distributed training strategy (sync, async etc) influence on model accuracy.
sort out differences between multi-machine-single-device and multi-machine-multi-device
better integration with single-machine training
think about more flexible user-customized device placement for multi-machine training.

[panyx0718]

fault-tolerance is a basic distributed training feature that probably doesn't belong to EDL only.

[seiriosPlus]

checkpoint need to be added to train feature.

[typhoonzero]

Maybe we should devide fault-tolerance to several parts:

• Base features
  • checkpointing and recovering on pserver
  • trainer pull checkpoint from pserver
  • recover reader offset (requires master and etcd)
• Clustering feature
  • autostart failed trainer based on cluster system (Kubernetes etc.)
  • autoscale trainer based on cluster system.

[typhoonzero]

The overall future roadmap should include the following parts:

1. Complete features of fluid distributed

• code clean up and polish
• implement LARS -- @typhoonzero doing
• pserver checkpointing
• init trainer weights from pserver
• distributed lookup table
• full overlapping with parallel executor and dist training
• complete async training, pserver use parallel executor
• remote executor runs ProgramDesc (depend on "Complete Fluid")

1. Able switch between communication libraries for different use cases.

• grpc performance enhancement
• OpenMPI with RDMA and GPU Direct
• NCCL2 with multi-node implement
• follow up brpc

1. EDL

• master implement
• etcd operators

1. CE

[Yancey1989]

Thanks, @panyx0718 @seiriosPlus @typhoonzero, I updated this issue followed by your comments.

[gongweibao]

Do we need design communication backend's abstract interface to be compatible with various implementations:

• Sync: nccl, mpi...
• Async: RPC

[gongweibao]

I think that it's maybe many things to do and we'd better do them with orders, classification, and priority.

[typhoonzero]

Closing this issue, most of the work are done except brpc and EDL related.