[SPARK-4281][Build] Package Yarn shuffle service into its own jar #3147

andrewor14 · 2014-11-07T05:12:07Z

This is another addendum to #3082, which added the Yarn shuffle service to run inside the NM. This PR makes the feature much more usable by packaging enough dependencies into the jar to run the service inside an NM. After these changes, the user can run ./make-distribution.sh and find a spark-network-yarn*.jar in their lib directory. The equivalent change is done in SBT by making the network-yarn module an assembly project.

This allows make-distribution to create a small uber jar for the network-yarn module, such that all uses of the Yarn shuffle service can just drop this jar onto the NM classpath and start the shuffle service after configuring the NM to include it.

SparkQA · 2014-11-07T05:20:01Z

Test build #23037 has started for PR 3147 at commit abcefd1.

This patch merges cleanly.

andrewor14 · 2014-11-07T05:22:01Z

I have tested the changes in both maven and SBT.
@pwendell Can you review the changes here?

SparkQA · 2014-11-07T06:43:29Z

Test build #23037 has finished for PR 3147 at commit abcefd1.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class StandaloneWorkerShuffleService(sparkConf: SparkConf, securityManager: SecurityManager)
- public class RetryingBlockFetcher

AmplabJenkins · 2014-11-07T06:43:32Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23037/
Test PASSed.

aarondav · 2014-11-07T06:51:13Z

network/common/pom.xml

@@ -41,12 +41,12 @@
      <groupId>io.netty</groupId>
      <artifactId>netty-all</artifactId>
    </dependency>
+
+    <!-- Provided dependencies -->
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-api</artifactId>


Why did you move the comment up here? Should this be at the provided scope?

Yeah, actually Yarn already provides slf4j so it doesn't need to be a core dependency. For standalone mode, this is also already required by Spark so it doesn't need to be a core dependency there either. HOWEVER I just realized I forgot to actually make it provided by adding the tag.

Ok I added the tag

SparkQA · 2014-11-07T17:29:55Z

Test build #23056 has started for PR 3147 at commit 65db822.

This patch merges cleanly.

SparkQA · 2014-11-07T18:56:53Z

Test build #23056 has finished for PR 3147 at commit 65db822.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-07T18:56:57Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23056/
Test PASSed.

pwendell · 2014-11-08T07:38:23Z

network/yarn/pom.xml

+        <artifactId>maven-shade-plugin</artifactId>
+        <configuration>
+          <shadedArtifactAttached>false</shadedArtifactAttached>
+          <outputFile>${project.build.directory}/scala-${scala.binary.version}/spark-network-yarn-${project.version}-hadoop${hadoop.version}.jar</outputFile>


Should we name this something like spark-yarn-shuffle-hadoop${hadoop.version}? The current name is very generic and, unlike our internal build, this will be user-facing since some folks might need to actually copy this jar into a location for YARN. It might be good to make it very obvious what this is.

Also - if this is going to be compatible across multiple YARN versions, maybe we should actually just put the Spark version instead: spark-${project.version}-yarn-shuffle.

pwendell · 2014-11-08T07:43:40Z

Hey Andrew - this looks good. I added some comment, all were regarding how we name the produced jar.

SparkQA · 2014-11-11T04:20:05Z

Test build #23191 has started for PR 3147 at commit bda58d0.

This patch merges cleanly.

SparkQA · 2014-11-11T05:42:25Z

Test build #23191 has finished for PR 3147 at commit bda58d0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-11T05:42:29Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23191/
Test PASSed.

pwendell · 2014-11-12T21:38:57Z

LGTM

andrewor14 · 2014-11-12T21:39:28Z

Ok, I merge into master and 1.2

pwendell · 2014-11-12T21:54:37Z

Okay, you merge

This is another addendum to #3082, which added the Yarn shuffle service to run inside the NM. This PR makes the feature much more usable by packaging enough dependencies into the jar to run the service inside an NM. After these changes, the user can run `./make-distribution.sh` and find a `spark-network-yarn*.jar` in their `lib` directory. The equivalent change is done in SBT by making the `network-yarn` module an assembly project. Author: Andrew Or <[email protected]> Closes #3147 from andrewor14/yarn-shuffle-build and squashes the following commits: bda58d0 [Andrew Or] Fix line too long 81e9705 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-shuffle-build fb7f398 [Andrew Or] Rename jar to spark-{VERSION}-yarn-shuffle.jar 65db822 [Andrew Or] Actually mark slf4j as provided abcefd1 [Andrew Or] Do the same for SBT c653028 [Andrew Or] Package network-yarn and its dependencies (cherry picked from commit aa43a8d) Signed-off-by: Andrew Or <[email protected]>

JoshRosen · 2014-11-13T02:57:07Z

I ran the make-distribution.sh script and it complained about being unable to find the YARN shuffle jar:

[...]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:30 min
[INFO] Finished at: 2014-11-12T18:53:57-08:00
[INFO] Final Memory: 58M/564M
[INFO] ------------------------------------------------------------------------
cp: /Users/joshrosen/Documents/Spark/network/yarn/target/scala*/spark-*-yarn-shuffle.jar: No such file or directory

arahuja · 2014-11-13T03:08:51Z

I had that as well when I did provide -Pyarn to make-distribution.sh, but it works when that is provided

andrewor14 · 2014-11-13T18:57:28Z

Good catch. The jar won't be there unless -Pyarn is provided. I'll fix this shortly.

andrewor14 · 2014-11-13T19:56:02Z

Alright the hot fix is merged. Thanks for reporting.

This is introduced in #3147 and is failing builds without the `-Pyarn` profile. Author: Andrew Or <[email protected]> Closes #3250 from andrewor14/fix-yarn-shuffle-build and squashes the following commits: 42b3d37 [Andrew Or] Do not fail fast if Yarn shuffle jar does not exist (cherry picked from commit a0fa1ba) Signed-off-by: Andrew Or <[email protected]>

This is introduced in #3147 and is failing builds without the `-Pyarn` profile. Author: Andrew Or <[email protected]> Closes #3250 from andrewor14/fix-yarn-shuffle-build and squashes the following commits: 42b3d37 [Andrew Or] Do not fail fast if Yarn shuffle jar does not exist

Andrew Or added 2 commits November 6, 2014 21:04

Do the same for SBT

abcefd1

andrewor14 changed the title ~~[SPARK-4281] Package Yarn shuffle service into its own jar~~ [SPARK-4281][Build] Package Yarn shuffle service into its own jar Nov 7, 2014

aarondav reviewed Nov 7, 2014
View reviewed changes

Actually mark slf4j as provided

65db822

pwendell reviewed Nov 8, 2014
View reviewed changes

Andrew Or added 3 commits November 10, 2014 20:12

Rename jar to spark-{VERSION}-yarn-shuffle.jar

fb7f398

Merge branch 'master' of github.com:apache/spark into yarn-shuffle-build

81e9705

Fix line too long

bda58d0

andrewor14 force-pushed the yarn-shuffle-build branch from 66a7868 to bda58d0 Compare November 11, 2014 04:16

asfgit closed this in aa43a8d Nov 12, 2014

andrewor14 deleted the yarn-shuffle-build branch November 13, 2014 01:26

andrewor14 mentioned this pull request Nov 13, 2014

[HOT FIX] make-distribution.sh fails if Yarn shuffle jar DNE #3250

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-4281][Build] Package Yarn shuffle service into its own jar #3147

[SPARK-4281][Build] Package Yarn shuffle service into its own jar #3147

andrewor14 commented Nov 7, 2014

SparkQA commented Nov 7, 2014

andrewor14 commented Nov 7, 2014

SparkQA commented Nov 7, 2014

AmplabJenkins commented Nov 7, 2014

aarondav Nov 7, 2014

andrewor14 Nov 7, 2014

andrewor14 Nov 7, 2014

SparkQA commented Nov 7, 2014

SparkQA commented Nov 7, 2014

AmplabJenkins commented Nov 7, 2014

pwendell Nov 8, 2014

pwendell Nov 8, 2014

pwendell commented Nov 8, 2014

SparkQA commented Nov 11, 2014

SparkQA commented Nov 11, 2014

AmplabJenkins commented Nov 11, 2014

pwendell commented Nov 12, 2014

andrewor14 commented Nov 12, 2014

pwendell commented Nov 12, 2014

JoshRosen commented Nov 13, 2014

arahuja commented Nov 13, 2014

andrewor14 commented Nov 13, 2014

andrewor14 commented Nov 13, 2014

[SPARK-4281][Build] Package Yarn shuffle service into its own jar #3147

[SPARK-4281][Build] Package Yarn shuffle service into its own jar #3147

Conversation

andrewor14 commented Nov 7, 2014

SparkQA commented Nov 7, 2014

andrewor14 commented Nov 7, 2014

SparkQA commented Nov 7, 2014

AmplabJenkins commented Nov 7, 2014

aarondav Nov 7, 2014

Choose a reason for hiding this comment

andrewor14 Nov 7, 2014

Choose a reason for hiding this comment

andrewor14 Nov 7, 2014

Choose a reason for hiding this comment

SparkQA commented Nov 7, 2014

SparkQA commented Nov 7, 2014

AmplabJenkins commented Nov 7, 2014

pwendell Nov 8, 2014

Choose a reason for hiding this comment

pwendell Nov 8, 2014

Choose a reason for hiding this comment

pwendell commented Nov 8, 2014

SparkQA commented Nov 11, 2014

SparkQA commented Nov 11, 2014

AmplabJenkins commented Nov 11, 2014

pwendell commented Nov 12, 2014

andrewor14 commented Nov 12, 2014

pwendell commented Nov 12, 2014

JoshRosen commented Nov 13, 2014

arahuja commented Nov 13, 2014

andrewor14 commented Nov 13, 2014

andrewor14 commented Nov 13, 2014