[MINOR][DOCS] Clarify that Spark apps should mark Spark as a 'provided' dependency, not package it

[srowen]

What changes were proposed in this pull request?

Spark apps do not need to package Spark. In fact it can cause problems in some cases. Our examples should show depending on Spark as a 'provided' dependency.

Packaging Spark makes the app much bigger by tens of megabytes. It can also bring in conflicting dependencies that wouldn't otherwise be a problem. https://issues.apache.org/jira/browse/SPARK-26146 was what reminded me of this.

How was this patch tested?

Doc build

[SparkQA]

Test build #102943 has finished for PR 23938 at commit f8fcc52.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[felixcheung]

all for it. could we add some explanation?

[dongjoon-hyun]

Could you update Accessing OpenStack Swift from Spark documentation, too?

• https://github.com/apache/spark/blame/master/docs/storage-openstack-swift.md#L33

[srowen]

@dongjoon-hyun yeah, that one I wasn't sure about as it's some support code that sounded like it was meant to be bundled in an app. @steveloughran is that correct -- hadoop-cloud should be a compile scope dependency, not provided by the cluster?

[steveloughran]

you should compile with hadoop-cloud and add those JARs it pulls in to the spark tarball placed on the shared cluster FS for YARN to pick up. Don't know about other deployment engines I'm afraid. The build also adds it to the SPARK_HOME/lib, which gives it to you for spark-standalone during spark submit, either for anything related to JAR upload, or for any store which implements delegation tokens (HADOOP-14456, HADOOP-16068, etc), so it collects the tokens for all stores listed in spark.yarn.hadoopFilesystems.

[srowen]

@steveloughran to be clear do you compile your app, or Spark, with this dependency? it sounds like "Spark" not the app. If so I'll update this further.

[steveloughran]

sorry, yeah, spark.

Even if the spark team doesn't redist those JARs, it'd be really useful if the release process published the POM. that way, if you want your build to pick up the exact set of dependencies which are in sync with spark, excluding all the stuff which will cause grief, you'd just add it as a dependency.

[srowen]

Ah OK on further review @steveloughran , the docs here are saying to include the dependency in your app, which would be the right thing if not bundled by Spark, and that's the current state of things for a default cluster. I think that much of the doc is then OK, and shouldn't change to mentioned provided.

[srowen]

Merged to master/2.4/2.3