[Rabit] One dmlc-core.

[trivialfis]

@chenqin @RAMitchell @hcho3

It was suggested that we can duplicate dmlc-core in #4352 but I disagree with the choice. The problem is in c++'s hairy linkage specification. If we have out-of-sync dmlc-core in two places, the behaviour is undefined. Specifically with two functions having the same signature linker will choose one of them and discard the other. I made a demonstration program for others to verify. Will upload soon.

[trivialfis]

Uploaded in https://github.com/trivialfis/xgbissues

[trivialfis]

We need to come up with a solution for the upgraded rabit.

[CodingCat]

Ok...then we should think about how to resolve the issue @RAMitchell raised

Two potential ways as initial thoughts:

1. Take the pain, update dmlc, upgrade submodule in Rabit, upgrade submodule(rabit) in XGBoost

2. let’s forget about rabit, keep rabit as an independent project for whoever else wants to use, but swallow everything from rabit to xgboost as an communication module

[trivialfis]

@CodingCat

For 1. Indeed painful.
For 2. If we fix a bug in our cloned rabit, we still want to port the fix to original rabit right? That's just copying files in disguise...

CMake has a feature called External project, that will fit normal development. But it also makes packaging for R CRAN (which uses amalgamation and requires source distribution) and Python PIP difficult.

[chenqin]

Rabit is a bit of everything but mostly flexible interface (sync/checkpoint/restore) Take an example, I can totally see running GPU direct implementation behind Rabit as possibility.

I think easiest thing we can do is to split dmlc-core headers used in rabit and rename (to avoid confusion in xgboost we had seen before) this will cut dependency between dmlc-core with rabit at cost of rabit needs to sync with dmlc-core from time to time to avoid xgboost rabit upgrade issue. It leaves rabit a bit more work and makes entire flow smooth.

The downside is clearly if dmlc-core headers changed like io.h rabit can only find out when it upgrade.(I think i can do a dummy script and auto sync with dmlc-core master with a bit naming hacking.

[CodingCat]

@trivialfis I am actually proposing accept more and more diversity of cloned rabit and the original rabit, if people wants to fix there, fine..and we can do whatever beneficial to xgboost in, as an example, src/communication/

[trivialfis]

@CodingCat I think @chenqin is doing a lots of work on rabit that I cannot do myself, and it seems jvm package always has tons of work to occupy your hand, so merging rabit into XGBoost won't advance it. My suggestion is add an option to rabit's build system for using existing dmlc-core instead of another git submodule.

[chenqin]

per comments in pr, by default built with xgboost dmlc-core, if use rabit without xgboost or mxnet it can choose to build with dmlc-core as subproject with make RABIT_BUID_DMLC=1

IMO, this can address duplicated code base while keep rabit always in sync with dmlc-core.

dmlc/rabit#91

[trivialfis]

Closed by dmlc/rabit#91 .