[EPIC] Allow failed worker retry in distributed training

[chenqin]

Fault recovery in native distributed XGB training (not xgb-spark) have been broken for more than a year. Community have seen various of issue like this dmlc/rabit#63

• expose flag to opt-in detailed log in rabit
• support bootstrap cache in rabit , expose flag to opt-in rabit bootstrap cache
  design doc.pdf
  PR candidate: support bootstrap allreduce/broadcast rabit#98
• patch native xgb code base and support worker retry
  pr remove is_bootstrap parameter rabit#102 cleanup rabit api
  pr [rabit_bootstrap_cache ] failed xgb worker recover from other workers #4808 patch xgb worker
  pr cover change with gtest unittests, clean up cmake script and code includes rabit#106 unittest rabit, cleanup make file
• patch xgb-spark code base and support task retry
  design [jvm-packages] retry-able xgboost4j-spark booster #4831
  • surface configuration to jvm layer [jvm-packages] expose rabit configuration to user #4861 pr [jvm-packages] update rabit, surface new changes to spark, add parity and failure tests #4966
  • short circuit, timeout rabit add timeout thread to avoid rabit hang forever rabit#105
  • enable failed foreachparition spark task retry and disable spark checkpoint if user opt-in with configuration "rabit_bootstrap_cache=1"

tracking backlogs work from merged prs

• move histogram init before loadcheckpoint. [rabit_bootstrap_cache ] failed xgb worker recover from other workers #4808 (comment)
• remove duplicated colmn size check in train/eval datasets
• cleanup duplicated rabit cmake build in xgboost pr cover change with gtest unittests, clean up cmake script and code includes rabit#106

[trivialfis]

Looks interesting. ;-)

[trams]

This looks interesting indeed. One question: "What is native distributed XGB training?" How can I use it? Is it the one which involves a python package + dask?

[chenqin]

This looks interesting indeed. One question: "What is native distributed XGB training?" How can I use it? Is it the one which involves a python package + dask?

If you try xgboost with more than one machine. You are already using rabit, we added feature and track changes in progress to xgb layer

https://www.slideshare.net/ChenQin1/scaling-xgboost

[trams]

@chenqin, I understand that xgboost uses rabit as AllReduce implementation. I still don't really know what is a native distributed XGB training?
Does xgboost-spark run native distributed XGB training? If yes then why do we add "native" here? What other distributed XGB training do we have?

I am new to the project and I am still discovering different ways to use xgboost so this is an honest question.

[chenqin]

@chenqin, I understand that xgboost uses rabit as AllReduce implementation. I still don't really know what is a native distributed XGB training?
Does xgboost-spark run native distributed XGB training? If yes then why do we add "native" here? What other distributed XGB training do we have?

I am new to the project and I am still discovering different ways to use xgboost so this is an honest question.

In some use cases where user don't have complex feature generating needs, user can launch xgboost native worker (c++) without data processing
framework. https://github.com/kubeflow/xgboost-operator

[chenqin]

from description in this thread, yes, dask-xgboost leverage rabit as well.
#2032

[hcho3]

@chenqin It's epic :)

[nateagr]

Hi there! Any update on this epic ? Thanks.

[chenqin]

Hi there! Any update on this epic ? Thanks.

Yes, it's going (holiday session slow), still miss the final piece of patch XGBoost-spark after we can get current open pr landed.

[trivialfis]

I don't think this is possible in short term. Let's stick with fail all strategy for now. We @hcho3 are thinking about redesigning RABIT from scratch.