Skip to content

Commit

Permalink
SPARK-7481: artifact is spark-hadoop-cloud while project is hadoop-cl…
Browse files Browse the repository at this point in the history
…oud. Added an extra callout to the docs "without some form of consistency layer, Amazon S3 cannot

be safely used as the direct destination of work with the normal rename-based committer"

That holds for all Spark-in-EC deployments; EMR has a consistency option, as do/will others; different committers can work reliably.

Change-Id: Ibbf8b1b9de10b5bb83b647cc663ceb970f70ff2d
Testing: SBT and maven, with & without the hadoop-2.7 option.
  • Loading branch information
steveloughran committed May 4, 2017
1 parent b788494 commit e173e3f
Show file tree
Hide file tree
Showing 3 changed files with 5 additions and 2 deletions.
2 changes: 1 addition & 1 deletion assembly/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,7 @@
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>hadoop-cloud_${scala.binary.version}</artifactId>
<artifactId>spark-hadoop-cloud_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
Expand Down
3 changes: 3 additions & 0 deletions docs/cloud-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,9 @@ For these reasons, it is not always safe to use an object store as a direct dest
an intermediate store in a chain of queries. Consult the documentation of the object store and its
connector to determine which uses are considered safe.

In particular: *without some form of consistency layer, Amazon S3 cannot
be safely used as the direct destination of work with the normal rename-based committer.*

### Installation

With the relevant libraries on the classpath and Spark configured with valid credentials,
Expand Down
2 changes: 1 addition & 1 deletion hadoop-cloud/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
<relativePath>../pom.xml</relativePath>
</parent>

<artifactId>hadoop-cloud_2.11</artifactId>
<artifactId>spark-hadoop-cloud_2.11</artifactId>
<packaging>jar</packaging>
<name>Spark Project Cloud Integration</name>
<description>
Expand Down

0 comments on commit e173e3f

Please sign in to comment.