Merge Hadoop Into Spark

[pwendell]

This patch merges the Hadoop 0.20.2 source code into the Spark project. I've thought about this a bunch and this will provide us with several benefits:

More source code

Let's be honest, to be taken seriously as a project Spark needs to have way more lines of code. Spark is currently 70,000 lines of Scala code - this patch adds 452,000 lines of XML alone (!) This will make our github stats look great!

Seamless builds

Sometimes users stumble trying to build Spark against Hadoop. Not anymore!! With Hadoop inside of Spark this won't a problem at all. I mean, there's basically only one Hadoop version, right? So this should work for pretty much everyone. Wait, hold on, is hadoop-0.20.2 the same as hadoop-2.2.0? I'm assuming it's the same because they both have the same number of "2"s.

Your favorite old configs

This patch will give users access to some of their favorite old configs from Hadoop. Did you just figure out what dfs.namenode.path.based.cache.block.map.allocation.percent was?! Now you can use it in Spark!! Pining for your old friend mapreduce.map.skip.proc.count.autoincr... fuggedabodit - we got ya!

I plan to contribute tests and docs in a subsequent patch. Please merge this ASAP and include in Spark 0.9.1.

NB: This diff is too large for github to render. Users will have to download and play with this on their own.

[rxin]

+1 !!! I have been asking for this multiple times on the mailing list and finally see the light!!!

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13636/

[aarondav]

But will this work on YARN?

[mengxr]

The best way to have more lines than some project is to merge it!

[shivaram]

I love hadoop-0.20.2 -- It is the best Hadoop I have ever used. Thanks @pwendell for pulling this in.

[mridulm]

LGTM, merged !

On Wed, Apr 2, 2014 at 12:20 AM, Shivaram Venkataraman <
[email protected]> wrote:

I love hadoop-0.20.2 -- It is the best Hadoop I have ever used. Thanks
@pwendell https://github.com/pwendell for pulling this in.

Reply to this email directly or view it on GitHubhttps://github.com//pull/286#issuecomment-39243472
.

[andrewor14]

LGTMT

[CodingCat]

Great job!

[sryza]

I'm OK with holding off on this for a separate JIRA, but for completeness I'd like to propose merging in CDH code as well. I believe that CDH4.2.2 would be the best choice of version, as it maintains 2-compatibility with 0.20.2 and 2.2.0.

[aarondav]

Guys, I just realized that this includes a really cool submodule called "Map Reduce" which seems to be a generalized form of Spark! It also has around 3x the amount of code, and that's in Java (which is somewhat more concise than Scala), so I estimate it's roughly 5x as awesome as Spark. Are there plans for deprecating the Spark API in favor of "Map Reduce"?

If we deprecate Spark in 0.9.1 and replace it with Map Reduce in 1.0.0, that should give users enough time to migrate to the new API (which is much simpler -- only two functions!).

[mattf]

+1, lgtm, but only today!

[mateiz]

Wait, so did it pass or fail Jenkins tests? Jenkins isn't saying what happened, only that the build finished.

Anyway I'd be okay with this as long as you make it meet our code style guidelines.

[CrazyJvm]

amazing! I've been looking forward it for a long time!

[DavyLin]

I do not know, ah, but I'm looking forward to it

[liancheng]

This is a good start! Why not merge all transitive dependencies into Spark altogether? In this way we can save lots of efforts on SBT/Maven, and avoid time consuming Ivy dependency resolution.

You know, ever since those .orbit things were added, sbt update costs nearly 40 minutes to resolve all the dependencies in mainland China... I have to turn it off by skip in update := true and offline := true.

OK, the 2nd paragraph is not part of the joke :-)

[yinxusen]

Yep, I find that each time I do sbt clean gen-idea or sbt update or even sbt testOnly xxx, I can do the cooking, take a shower, and have a rest.

[pelick]

Is it true about this => “ Wait, hold on, is hadoop-0.20.2 the same as hadoop-2.2.0? I'm assuming it's the same because they both have the same number of "2"s. ”

[win2cs]

"But will this work on YARN?", I've the same question.
And does it still support specifying other Hadoop distribution like before?

[zsxwing]

You know, ever since those .orbit things were added, sbt update costs nearly 40 minutes to resolve all the dependencies in mainland China... I have to turn it off by skip in update := true and offline := true.

+1. Really terrible nightmare. If the version of some jar is 6.4, it usually can not be downloaded in mainland China without a proxy :(