Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

support bootstrap allreduce/broadcast #98

Merged
merged 95 commits into from
Aug 28, 2019
Merged

support bootstrap allreduce/broadcast #98

merged 95 commits into from
Aug 28, 2019

Conversation

chenqin
Copy link
Contributor

@chenqin chenqin commented Jul 2, 2019 •
edited
Loading

The goal of this pr is to implement immutable cache in rabit to help failed worker recover not syned allreduces in bootstrap time. examples of such are

  • distributed xgboost load dataset and sync number of columns before loadcheckpoint.
  • init histgram in fast hist algorithm only runs before completion of first iteration checkpoint

It's specifically designed to help recovered node catch up with rest of nodes with minimal overhead in non recovery mode.

  • setcache only writes locally after allreduce/broadcast complete
  • getcache send/recv two integer in one allreduce call and decide if all nodes are running in sync
  • when nodes are in sync, skip reading cache, only recovery nodes in bootstrap do one time catch rebuild from nearest nodes

in addational, this pr also includes

  • rabit_cache flag to opt-in this feature
  • rabit_debug flag to opt-in allreduce/broadcast/loadcheckpoint/checkpoint operation detail log for debugging usage

design doc

Chen Qin and others added 29 commits June 10, 2019 14:00
support run rabit tests as xgboost subproject using xgboost/dmlc-core
ed06620
@chenqin
support tracker config set/get
dddcac7
@chenqin
remove redudant printf
2fac91b
@chenqin
remove redudant printf
f5a9727
@chenqin
Merge branch 'master' of https://github.com/chenqin/rabit
acc011d
@chenqin
add c++0x declaration
e391238
@chenqin
log allreduce/broadcast caller, engine should track caller stack for
a9d7331 
investigation
@chenqin
tracker support binary config format
2a28e5e
@chenqin
Revert "tracker support binary config format"
2c322f3 
This reverts commit 2a28e5e.
@chenqin
remove caller, prototype fetch allreduce/broadcast results from resbuf
19ce2de
@chenqin
store cached allreduce/broadcast seq_no to tracker
4d0a1e1
@chenqin
allow restore all caches from other nodes
35bcd61
@chenqin
try new rabit collective cache, todo: recv_link seems down
68135c8
@chenqin
link up cache restore with main recovery
87c62ea
@chenqin
cleanup load cache state
05d5f01
update cache api
2f95628
@chenqin
pass test.mk
1178bac
@chenqin
have a working tests
435f681
@chenqin
try to unify check into actionsummary
22571a2
more logging to debug distributed hist three method issue
809181c
@chenqin
update rabit interface to support caller signature matching
289d916
@chenqin
splite seq_counter from cur_cache_seq to different variables
c033576
@chenqin
still see issue with inf loop
95330c6
@chenqin
support debug print caller as well as allreduce op
73a6746
cleanup
337b447
@chenqin
remove get/set cache from model_recover, adding recover in
ee234c4 
loadcheckpoint
@chenqin
clarify rabit cache strategy, cache is set only by successful collective
ef235c4 
call involving all nodes with unique cache key. if all nodes call
getcache at same time, we keep rabit run collective call. If some nodes
call getcache while others not, we backfill cache from those nodes with
most entries
revert caller logs
98380bc
fix lint error
db343f9
@chenqin chenqin marked this pull request as ready for review July 2, 2019 22:46
@chenqin
Copy link
Contributor Author

chenqin commented Aug 19, 2019

@hcho3

@hcho3
Copy link
Contributor

hcho3 commented Aug 20, 2019

@chenqin Can we simply throw error when the user is not using Linux and enables bootstrapping? We can add non-Linux support later.

@chenqin
Copy link
Contributor Author

chenqin commented Aug 22, 2019 •
edited
Loading

@chenqin Can we simply throw error when the user is not using Linux and enables bootstrapping? We can add non-Linux support later.

Per convo offline, it still works even with env not supporting _buildin_func, thanks to the fact that we also use type_size and count as part of signature as well.

image

If there is a collision of signature as we change implementation or adding new tree method we could do

  1. explicit change default value and avoid colision
    image

  2. if we forgot to do so, it will trigger assertion error
    image

So it is still safe

CodingCat
CodingCat reviewed Aug 23, 2019
View reviewed changes
src/allreduce_base.cc Outdated Show resolved Hide resolved
@chenqin
Copy link
Contributor Author

chenqin commented Aug 26, 2019 •
edited
Loading

are we comfortable to merge this back to master? @hcho3

CodingCat
CodingCat reviewed Aug 26, 2019
View reviewed changes
CMakeLists.txt Outdated Show resolved Hide resolved
CodingCat
CodingCat reviewed Aug 26, 2019
View reviewed changes
src/allreduce_base.cc Outdated Show resolved Hide resolved
src/allreduce_robust.h Outdated Show resolved Hide resolved
@CodingCat
Copy link
Member

so I have left all my comments here, the PR has improved a lot since I put change requested previously, but GitHub doesn't allow me to change my opinion and I feel that I am not 100% confident on reviewing all tech details here. I will defer to @hcho3 for final decision

@chenqin
Copy link
Contributor Author

chenqin commented Aug 27, 2019

going to asking around @tqchen @CodingCat @hcho3 @trivialfis can we merge this?

@hcho3 hcho3 merged commit 5797dcb into dmlc:master Aug 28, 2019
@hcho3
Copy link
Contributor

hcho3 commented Aug 28, 2019

Done

@chenqin chenqin mentioned this pull request Aug 30, 2019
Merged
@hcho3 hcho3 mentioned this pull request Sep 2, 2019
Closed
@chenqin
Copy link
Contributor Author

chenqin commented Sep 5, 2019

removed is_bootstrap parameter in upcoming pr
69789ec

@CodingCat CodingCat mentioned this pull request Nov 6, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CodingCat CodingCat CodingCat requested changes

@hcho3 hcho3 hcho3 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@chenqin @CodingCat @tqchen @hcho3