[SPARK-5135][SQL] Add support for describe [extended] table to DDL in SQLContext #3935

OopsOutOfMemory · 2015-01-07T17:06:49Z

RT.
issue is here : https://issues.apache.org/jira/browse/SPARK-5135

AmplabJenkins · 2015-01-07T17:07:10Z

Can one of the admins verify this patch?

OopsOutOfMemory · 2015-01-07T17:22:14Z

e.g.

import org.apache.spark.sql.SQLContext
val sqlContext  = new SQLContext(sc) 
import sqlContext._  

val jsonDDL = s"""  
      |CREATE TEMPORARY TABLE jsonTable  
      |USING org.apache.spark.sql.json  
      |OPTIONS (  
      | path  'file:///Users/shengli/git_repos/spark/examples/src/main/resources/people.json'  
      |)""".stripMargin

sqlContext.sql(jsonDDL)

sqlContext.sql("describe jsonTable").collect()

sqlContext.sql("describe extended jsonTable").collect()

scala> sqlContext.sql("describe jsonTable").collect()
15/01/08 01:20:11 INFO sources.DDLParser: is extended ? -> false
res1: Array[org.apache.spark.sql.Row] = Array([age,int,null], [name,string,null])

scala> 

scala> sqlContext.sql("describe extended jsonTable").collect()
15/01/08 01:20:14 INFO sources.DDLParser: is extended ? -> true
res2: Array[org.apache.spark.sql.Row] = Array([age,int,null], [name,string,null], [# extended,null,null])

marmbrus · 2015-01-07T23:24:26Z

ok to test

marmbrus · 2015-01-07T23:24:36Z

you'll also need to add test cases

SparkQA · 2015-01-07T23:27:34Z

Test build #25179 has started for PR 3935 at commit 1c02744.

This patch merges cleanly.

SparkQA · 2015-01-07T23:28:26Z

Test build #25179 has finished for PR 3935 at commit 1c02744.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DDLStrategy(context: SQLContext) extends Strategy
- case class DDLDescribeCommand(

AmplabJenkins · 2015-01-07T23:28:27Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25179/
Test FAILed.

chenghao-intel · 2015-01-08T00:51:59Z

sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala

@@ -328,18 +328,19 @@ class SQLContext(@transient val sparkContext: SparkContext)
    def numPartitions = self.numShufflePartitions

    def strategies: Seq[Strategy] =
-      extraStrategies ++ (


Nit: a :: b :: c :: Nil is exact the same with Seq(a, b, c). :)

yeah, I just make this same as hive context way.
@chenghao-intel
This test failed with Refer to this link for build results (access rights to CI server needed):
no rights to CL server ?

Seems a bug of error reporting, anyway, the root reason is it failed in the code style checking.
/home/jenkins/workspace/SparkPullRequestBuilder/sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala:198:21: Insert a space after the start of the comment

Can we revert SQLContext in this PR? Since the change is equivalent.

I think it's ok, it can at least save some space since :: is 2 chars, , is 1 char.

As @chenghao-intel said, do you mind not changing this part? It would be great if the PR focuses on what it does.

@chenghao-intel @rxin
ok, got it. I will change this.

chenghao-intel · 2015-01-08T01:13:34Z

That's a nice feature in general, I agree with @marmbrus , we do need a test suite for this.

OopsOutOfMemory · 2015-01-08T03:48:54Z

@chenghao-intel @marmbrus
Added test cases.

OopsOutOfMemory · 2015-01-08T03:48:59Z

ok to test

SparkQA · 2015-01-08T03:52:34Z

Test build #25198 has started for PR 3935 at commit d0fe2d6.

This patch merges cleanly.

SparkQA · 2015-01-08T05:00:16Z

Test build #25198 has finished for PR 3935 at commit d0fe2d6.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DDLStrategy(context: SQLContext) extends Strategy
- case class DDLDescribeCommand(

AmplabJenkins · 2015-01-08T05:00:19Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25198/
Test PASSed.

chenghao-intel · 2015-01-08T08:36:32Z

sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala

+
+  override def run(sqlContext: SQLContext) = {
+    val rows = new ArrayBuffer[Row]()
+    rows += Row("# col_name", "data_type", "comment")


"# col_name", "data_type", "comment" is the output field name, we'd better not take that as part of the output.

@chenghao-intel
ok, I will remove it.
If future support partition table here, I will add it like hive does:

# Partition Information # col_name data_type comment patition_col_name string None

SparkQA · 2015-01-08T09:27:32Z

Test build #25216 has started for PR 3935 at commit fbe6f6d.

This patch merges cleanly.

SparkQA · 2015-01-08T10:35:34Z

Test build #25216 has finished for PR 3935 at commit fbe6f6d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-08T10:35:37Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25216/
Test PASSed.

SparkQA · 2015-01-08T15:02:37Z

Test build #25226 has started for PR 3935 at commit 47f5593.

This patch merges cleanly.

SparkQA · 2015-01-08T15:03:30Z

Test build #25226 has finished for PR 3935 at commit 47f5593.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DDLDescribeCommand(dbName: Option[String], tableName: String, isExtended: Boolean) extends RunnableCommand

AmplabJenkins · 2015-01-08T15:03:31Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25226/
Test FAILed.

OopsOutOfMemory · 2015-01-08T15:04:35Z

@scwf @chenghao-intel
I've changed the implementation to RunnableCommand and removed DDLStrategy.
Could u review it ?

OopsOutOfMemory · 2015-01-08T15:04:46Z

test this please.

SparkQA · 2015-01-08T15:07:38Z

Test build #25227 has started for PR 3935 at commit f5821ae.

This patch merges cleanly.

SparkQA · 2015-01-08T15:17:32Z

Test build #25229 has started for PR 3935 at commit aa34164.

This patch merges cleanly.

SparkQA · 2015-01-16T07:27:41Z

Test build #25639 has started for PR 3935 at commit 521bbd7.

This patch merges cleanly.

SparkQA · 2015-01-16T07:30:53Z

Test build #25639 has finished for PR 3935 at commit 521bbd7.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DDLDescribeCommand(

AmplabJenkins · 2015-01-16T07:30:55Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25639/
Test FAILed.

OopsOutOfMemory · 2015-01-16T08:02:23Z

@rxin The rebase may have some problems, cause changed 278 files ? how do I revert it ?

rxin · 2015-01-16T08:13:31Z

Not sure - do you have a backup? Maybe just take a diff and apply the diff on master.

SparkQA · 2015-01-16T09:27:43Z

Test build #25651 has started for PR 3935 at commit 9efdf35.

This patch merges cleanly.

SparkQA · 2015-01-16T09:36:56Z

Test build #25651 has finished for PR 3935 at commit 9efdf35.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DDLDescribeCommand(

AmplabJenkins · 2015-01-16T09:36:59Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25651/
Test FAILed.

SparkQA · 2015-01-16T10:07:42Z

Test build #25654 has started for PR 3935 at commit 9d22708.

This patch merges cleanly.

scwf · 2015-01-16T10:12:13Z

@OopsOutOfMemory, you need revert unnecessary changes

SparkQA · 2015-01-16T10:52:42Z

Test build #25655 has started for PR 3935 at commit 5b7ae19.

This patch merges cleanly.

OopsOutOfMemory · 2015-01-16T10:54:53Z

Thanks @scwf @rxin, conflicts resolved cleanly, this now up-to-date.

SparkQA · 2015-01-16T11:48:37Z

Test build #25654 has finished for PR 3935 at commit 9d22708.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DDLDescribeCommand(

AmplabJenkins · 2015-01-16T11:48:40Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25654/
Test PASSed.

SparkQA · 2015-01-16T12:02:33Z

Test build #25655 has finished for PR 3935 at commit 5b7ae19.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DDLDescribeCommand(

AmplabJenkins · 2015-01-16T12:02:36Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25655/
Test PASSed.

SparkQA · 2015-01-16T15:52:39Z

Test build #25665 has started for PR 3935 at commit d1689e2.

This patch merges cleanly.

SparkQA · 2015-01-16T17:00:56Z

Test build #25665 has finished for PR 3935 at commit d1689e2.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class DDLDescribeCommand(

AmplabJenkins · 2015-01-16T17:00:59Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25665/
Test PASSed.

rxin · 2015-01-21T00:30:32Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala

+      val ddlPlan = ddlParser(sqlText)
+      val basicPlan = try {
+        HiveQl.parseSql(sqlText)
+      }catch {


add a space here

actually it's ok i will fix it myself

[SPARK-5135][SQL] Add support for describe [extended] table to DDL in SQLContext Conflicts: sql/catalyst/src/main/scala/org/apache/spark/sql/types/dataTypes.scala

rxin · 2015-01-21T00:43:13Z

Thanks. I resolved the conflict and pushed a PR #4127

OopsOutOfMemory · 2015-01-21T02:08:26Z

Thanks @rxin
Should I close this PR?

rxin · 2015-01-21T03:01:33Z

Yea we can close this one. Will merge that one when tests pass.

chenghao-intel reviewed Jan 8, 2015
View reviewed changes

OopsOutOfMemory force-pushed the describe branch from 9efdf35 to 9d22708 Compare January 16, 2015 10:06

patch

5b7ae19

OopsOutOfMemory force-pushed the describe branch from 9d22708 to 5b7ae19 Compare January 16, 2015 10:47

refine imports

d1689e2

rxin reviewed Jan 21, 2015
View reviewed changes

rxin mentioned this pull request Jan 21, 2015

[SPARK-5135][SQL] Add support for describe [extended] table to DDL in SQLContext #4127

Closed

OopsOutOfMemory closed this Jan 21, 2015

[SPARK-5135][SQL] Add support for describe [extended] table to DDL in SQLContext #3935

[SPARK-5135][SQL] Add support for describe [extended] table to DDL in SQLContext #3935

Conversation

OopsOutOfMemory commented Jan 7, 2015

AmplabJenkins commented Jan 7, 2015

OopsOutOfMemory commented Jan 7, 2015

marmbrus commented Jan 7, 2015

marmbrus commented Jan 7, 2015

SparkQA commented Jan 7, 2015

SparkQA commented Jan 7, 2015

AmplabJenkins commented Jan 7, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chenghao-intel commented Jan 8, 2015

OopsOutOfMemory commented Jan 8, 2015

OopsOutOfMemory commented Jan 8, 2015

SparkQA commented Jan 8, 2015

SparkQA commented Jan 8, 2015

AmplabJenkins commented Jan 8, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jan 8, 2015

SparkQA commented Jan 8, 2015

AmplabJenkins commented Jan 8, 2015

SparkQA commented Jan 8, 2015

SparkQA commented Jan 8, 2015

AmplabJenkins commented Jan 8, 2015

OopsOutOfMemory commented Jan 8, 2015

OopsOutOfMemory commented Jan 8, 2015

SparkQA commented Jan 8, 2015

SparkQA commented Jan 8, 2015

SparkQA commented Jan 16, 2015

SparkQA commented Jan 16, 2015

AmplabJenkins commented Jan 16, 2015

OopsOutOfMemory commented Jan 16, 2015

rxin commented Jan 16, 2015

SparkQA commented Jan 16, 2015

SparkQA commented Jan 16, 2015

AmplabJenkins commented Jan 16, 2015

SparkQA commented Jan 16, 2015

scwf commented Jan 16, 2015

SparkQA commented Jan 16, 2015

OopsOutOfMemory commented Jan 16, 2015

SparkQA commented Jan 16, 2015

AmplabJenkins commented Jan 16, 2015

SparkQA commented Jan 16, 2015

AmplabJenkins commented Jan 16, 2015

SparkQA commented Jan 16, 2015

SparkQA commented Jan 16, 2015

AmplabJenkins commented Jan 16, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rxin commented Jan 21, 2015

OopsOutOfMemory commented Jan 21, 2015

rxin commented Jan 21, 2015