[SPARK-17932][SQL] Support SHOW TABLES EXTENDED LIKE 'identifier_with_wildcards' statement #15958

jiangxb1987 · 2016-11-21T07:46:56Z

What changes were proposed in this pull request?

Currently we haven't implemented SHOW TABLE EXTENDED in Spark 2.0. This PR is to implement the statement.
Goals:

Support SHOW TABLES EXTENDED LIKE 'identifier_with_wildcards';
Explicitly output an unsupported error message for SHOW TABLES [EXTENDED] ... PARTITION statement;
Improve test cases for SHOW TABLES statement.

How was this patch tested?

Add new test cases in file show-tables.sql.
Modify tests for SHOW TABLES in DDLSuite.

SparkQA · 2016-11-21T07:47:32Z

Test build #68926 has started for PR 15958 at commit 89d4f5d.

jiangxb1987 · 2016-11-21T16:45:40Z

retest this please.

SparkQA · 2016-11-21T19:01:41Z

Test build #68943 has finished for PR 15958 at commit 89d4f5d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jiangxb1987 · 2016-11-22T10:38:08Z

cc @hvanhovell @gatorsmile

gatorsmile · 2016-11-22T23:25:40Z

cc @dilipbiswal The original code was wrote by you. Could you review it? Thanks!

dilipbiswal · 2016-11-23T01:46:29Z

@gatorsmile Thank you for cc'ing me. Overall it looks good to me.
I have a few observations/comments that i would list below which hopefully should help us
decide.

Hive actually has two commands i) show tables ii) show table and the extended
switch is applicable for the "show table" command. Have we decided to merge
the semantics of these two commands in an effort to similify ?
As a side effect, we actually print the database name and table name two times.
Is that ok ?
The format of hive seems better as there is a newline between two tables. I created
a regular table and a temporary view with same prefix and tried the command. Its very
difficult to distinguish the output between table and view. here is how it looks.

spark-sql> create table realtable(c1 int) using parquet;
Time taken: 0.467 seconds
spark-sql> create temporary view realview (c1 int, c2 int) using parquet;
Time taken: 0.053 seconds
spark-sql> show tables extended like 'real*';
default	realtable	false	CatalogTable(
	Table: `default`.`realtable`
	Owner: cloudera
	Created: Tue Nov 22 17:41:56 PST 2016
	Last Access: Wed Dec 31 16:00:00 PST 1969
	Type: MANAGED
	Schema: [StructField(c1,IntegerType,true)]
	Provider: parquet
	Properties: [transient_lastDdlTime=1479865316]
	Storage(Location: file:/user/hive/warehouse/realtable, InputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat, OutputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat, Serde: org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe, Properties: [serialization.format=1]))
	realview	true	CatalogTable(
	Table: `realview`
	Created: Tue Nov 22 17:42:28 PST 2016
	Last Access: Wed Dec 31 15:59:59 PST 1969
	Type: VIEW
	Schema: [StructField(c1,IntegerType,true), StructField(c2,IntegerType,true)]
	Storage())
Time taken: 0.104 seconds, Fetched 2 row(s)

jiangxb1987 · 2016-11-23T10:06:44Z

@dilipbiswal Thank you for reviewing this PR! I want to share my thoughts about the questions:

Perhaps using the same semantics is better IMO, because that'll be less comfusing. I've often mistype between show table and show tables in HIVE;
I'm neutral to the issue whether we should still print the database name and the table name when we have printed the extended information;
+1 for that the format should be improved, perhaps add one empty line after each row?

SparkQA · 2016-11-24T11:50:50Z

Test build #69128 has finished for PR 15958 at commit a3db00f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2016-11-27T00:14:59Z

Could you update the PR title? The current one is confusing. This PR is not a bug fix.

jiangxb1987 · 2016-11-27T06:12:06Z

@gatorsmile I've updated the title, thank you!

gatorsmile · 2016-11-27T06:15:19Z

The Non-Goals in the PR description is also confusing. Basically, what this PR does is to explicitly output an unsupported error message if users specify PARTITION. Could you also update that? Thanks!

hvanhovell · 2016-11-28T14:51:03Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala

  }

  override def run(sparkSession: SparkSession): Seq[Row] = {
+    if (isExtended && !tableIdentifierPattern.isDefined) {
+      throw new AnalysisException(
+        s"SHOW TABLES EXTENDED must have identifier_with_wildcards specified.")


Throw this exception when creating the class. This is to late.

SparkQA · 2016-11-30T07:57:41Z

Test build #69393 has started for PR 15958 at commit 958fe8b.

jiangxb1987 · 2016-11-30T08:18:53Z

retest this please.

SparkQA · 2016-11-30T11:00:00Z

Test build #69395 has finished for PR 15958 at commit 958fe8b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell · 2016-11-30T11:08:07Z

LGTM. Merging to master. Thanks!

gatorsmile · 2016-11-30T18:19:19Z

sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4

-    | SHOW TABLES ((FROM | IN) db=identifier)?
-        (LIKE? pattern=STRING)?                                        #showTables
+    | SHOW TABLES EXTENDED? ((FROM | IN) db=identifier)?
+        (LIKE? pattern=STRING)? partitionSpec?                         #showTables


This Syntax is different from Hive.

SHOW TABLE EXTENDED [IN|FROM database_name] LIKE 'identifier_with_wildcards' [PARTITION(partition_spec)];

gatorsmile · 2016-11-30T18:19:46Z

@jiangxb1987 Could you submit a follow-up PR to correct it? Thanks!

jiangxb1987 · 2016-12-01T02:11:56Z

@gatorsmile Could you explicitly point out which part should be modified? IMO we don't follow the HIVE syntax unconditionally. Thanks!

gatorsmile · 2016-12-01T02:37:40Z

If needed, we should not break the consistency. For example, SHOW TABLE and SHOW TABLES are different.

…_wildcards' statement ## What changes were proposed in this pull request? Currently we haven't implemented `SHOW TABLE EXTENDED` in Spark 2.0. This PR is to implement the statement. Goals: 1. Support `SHOW TABLES EXTENDED LIKE 'identifier_with_wildcards'`; 2. Explicitly output an unsupported error message for `SHOW TABLES [EXTENDED] ... PARTITION` statement; 3. Improve test cases for `SHOW TABLES` statement. ## How was this patch tested? 1. Add new test cases in file `show-tables.sql`. 2. Modify tests for `SHOW TABLES` in `DDLSuite`. Author: jiangxingbo <[email protected]> Closes apache#15958 from jiangxb1987/show-table-extended.

implement SHOW TABLES EXTENDED.

89d4f5d

add a empty line between show table extended output rows.

a3db00f

jiangxb1987 changed the title ~~[SPARK-17932][SQL] Failed to run SQL "show table extended like table_name" in Spark2.0.0~~ [SPARK-17932][SQL] Support SHOW TABLES EXTENDED LIKE 'identifier_with_wildcards' statement Nov 27, 2016

hvanhovell reviewed Nov 28, 2016

View reviewed changes

throw Exceptions in a early stage.

958fe8b

asfgit closed this in c24076d Nov 30, 2016

gatorsmile reviewed Nov 30, 2016

View reviewed changes

jiangxb1987 deleted the show-table-extended branch December 9, 2016 10:51

gatorsmile mentioned this pull request Dec 13, 2016

[SPARK-17932][SQL][FOLLOWUP] Change statement SHOW TABLES EXTENDED to SHOW TABLE EXTENDED #16262

Closed

beckjake mentioned this pull request Mar 27, 2020

dbt fails cryptically on old (I assume?) versions of spark dbt-labs/dbt-spark#71

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-17932][SQL] Support SHOW TABLES EXTENDED LIKE 'identifier_with_wildcards' statement #15958

[SPARK-17932][SQL] Support SHOW TABLES EXTENDED LIKE 'identifier_with_wildcards' statement #15958

jiangxb1987 commented Nov 21, 2016 •

edited

Loading

SparkQA commented Nov 21, 2016

jiangxb1987 commented Nov 21, 2016

SparkQA commented Nov 21, 2016

jiangxb1987 commented Nov 22, 2016

gatorsmile commented Nov 22, 2016

dilipbiswal commented Nov 23, 2016

jiangxb1987 commented Nov 23, 2016

SparkQA commented Nov 24, 2016

gatorsmile commented Nov 27, 2016

jiangxb1987 commented Nov 27, 2016

gatorsmile commented Nov 27, 2016

hvanhovell Nov 28, 2016

SparkQA commented Nov 30, 2016

jiangxb1987 commented Nov 30, 2016

SparkQA commented Nov 30, 2016

hvanhovell commented Nov 30, 2016

gatorsmile Nov 30, 2016

gatorsmile commented Nov 30, 2016

jiangxb1987 commented Dec 1, 2016

gatorsmile commented Dec 1, 2016

[SPARK-17932][SQL] Support SHOW TABLES EXTENDED LIKE 'identifier_with_wildcards' statement #15958

[SPARK-17932][SQL] Support SHOW TABLES EXTENDED LIKE 'identifier_with_wildcards' statement #15958

Conversation

jiangxb1987 commented Nov 21, 2016 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

SparkQA commented Nov 21, 2016

jiangxb1987 commented Nov 21, 2016

SparkQA commented Nov 21, 2016

jiangxb1987 commented Nov 22, 2016

gatorsmile commented Nov 22, 2016

dilipbiswal commented Nov 23, 2016

jiangxb1987 commented Nov 23, 2016

SparkQA commented Nov 24, 2016

gatorsmile commented Nov 27, 2016

jiangxb1987 commented Nov 27, 2016

gatorsmile commented Nov 27, 2016

hvanhovell Nov 28, 2016

Choose a reason for hiding this comment

SparkQA commented Nov 30, 2016

jiangxb1987 commented Nov 30, 2016

SparkQA commented Nov 30, 2016

hvanhovell commented Nov 30, 2016

gatorsmile Nov 30, 2016

Choose a reason for hiding this comment

gatorsmile commented Nov 30, 2016

jiangxb1987 commented Dec 1, 2016

gatorsmile commented Dec 1, 2016

jiangxb1987 commented Nov 21, 2016 •

edited

Loading