Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DOC] Adjust coverage for partitionBy() #10499

Closed
wants to merge 4 commits into from
Closed

[DOC] Adjust coverage for partitionBy() #10499

wants to merge 4 commits into from

Conversation

ted-yu
Copy link

@ted-yu ted-yu commented Dec 28, 2015

This is the related thread: http://search-hadoop.com/m/q3RTtO3ReeJ1iF02&subj=Re+partitioning+json+data+in+spark

Michael suggested fixing the doc.

Please review.

JoshRosen
JoshRosen reviewed Dec 28, 2015
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
@@ -119,7 +119,7 @@ final class DataFrameWriter private[sql](df: DataFrame) {
* Partitions the output by the given columns on the file system. If specified, the output is
* laid out on the file system similar to Hive's partitioning scheme.
*
* This is only applicable for Parquet at the moment.
* This was initally applicable for Parquet but in 1.5.x covers JSON as well.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1.5.x or 1.5+?

@SparkQA
Copy link

SparkQA commented Dec 28, 2015

Test build #48373 has finished for PR 10499 at commit 7884e87.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Dec 29, 2015

Test build #48381 has finished for PR 10499 at commit f655bbe.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

marmbrus
marmbrus reviewed Dec 29, 2015
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
@@ -119,7 +119,7 @@ final class DataFrameWriter private[sql](df: DataFrame) {
* Partitions the output by the given columns on the file system. If specified, the output is
* laid out on the file system similar to Hive's partitioning scheme.
*
* This is only applicable for Parquet at the moment.
* This was initally applicable for Parquet but in 1.5+ covers JSON as well.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also "text"

@SparkQA
Copy link

SparkQA commented Dec 29, 2015

Test build #48391 has finished for PR 10499 at commit dff3935.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

marmbrus
marmbrus reviewed Dec 29, 2015
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
@@ -119,7 +119,7 @@ final class DataFrameWriter private[sql](df: DataFrame) {
* Partitions the output by the given columns on the file system. If specified, the output is
* laid out on the file system similar to Hive's partitioning scheme.
*
* This is only applicable for Parquet at the moment.
* This was initially applicable for Parquet but in 1.5+ covers JSON as well.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And text, ORC, and avro

@SparkQA
Copy link

SparkQA commented Dec 30, 2015

Test build #48443 has finished for PR 10499 at commit a021725.

  • This patch fails from timeout after a configured wait of 250m.
  • This patch merges cleanly.
  • This patch adds no public classes.

@tedyu
Copy link
Contributor

tedyu commented Dec 30, 2015

The test failure was not related to the patch.
Looks like HiveThriftBinaryServerSuite timed out:

[info] HiveThriftBinaryServerSuite:
[info] - GetInfo Thrift API (431 milliseconds)
[info] - JDBC query execution (9 seconds, 740 milliseconds)
[info] - Checks Hive version (1 second, 386 milliseconds)
[info] - SPARK-3004 regression: result set containing NULL (1 second, 685 milliseconds)
[info] - SPARK-4292 regression: result set iterator issue (5 seconds, 77 milliseconds)
[info] - SPARK-4309 regression: Date type support (1 second, 328 milliseconds)
[info] - SPARK-4407 regression: Complex type support (2 seconds, 165 milliseconds)
[info] - test multiple session (5 seconds, 857 milliseconds)
Attempting to post to Github...
 > Post successful.
Build step 'Execute shell' marked build as failure

@tedyu
Copy link
Contributor

tedyu commented Dec 30, 2015

@marmbrus :
Is there anything I need to do ?

@tedyu
Copy link
Contributor

tedyu commented Jan 4, 2016

@marmbrus
Gentle ping

@marmbrus
Copy link
Contributor

marmbrus commented Jan 4, 2016

Thanks, merged to master and 1.6

asfgit pushed a commit that referenced this pull request Jan 4, 2016
@tedyu @marmbrus
[DOC] Adjust coverage for partitionBy()
1005ee3 
This is the related thread: http://search-hadoop.com/m/q3RTtO3ReeJ1iF02&subj=Re+partitioning+json+data+in+spark

Michael suggested fixing the doc.

Please review.

Author: tedyu <[email protected]>

Closes #10499 from ted-yu/master.

(cherry picked from commit 40d0396)
Signed-off-by: Michael Armbrust <[email protected]>
@asfgit asfgit closed this in 40d0396 Jan 4, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@ted-yu @SparkQA @tedyu @marmbrus @JoshRosen