[SPARK-30724][SQL] Support 'LIKE ANY' and 'LIKE ALL' operators #27477

wangyum · 2020-02-06T13:17:41Z

What changes were proposed in this pull request?

LIKE ANY/SOME and LIKE ALL operators are mostly used when we are matching a text field with numbers of patterns. For example:

Teradata / Hive 3.0 / Snowflake:

--like any
select 'foo' LIKE ANY ('%foo%','%bar%');

--like all
select 'foo' LIKE ALL ('%foo%','%bar%');

PostgreSQL:

-- like any
select 'foo' LIKE ANY (array['%foo%','%bar%']);

-- like all
select 'foo' LIKE ALL (array['%foo%','%bar%']);

This PR add support these two operators.

More details:
https://docs.teradata.com/reader/756LNiPSFdY~4JcCCcR5Cw/4~AyrPNmDN0Xk4SALLo6aQ
https://issues.apache.org/jira/browse/HIVE-15229
https://docs.snowflake.net/manuals/sql-reference/functions/like_any.html

Why are the changes needed?

To smoothly migrate SQLs to Spark SQL.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Unit test.

SparkQA · 2020-02-06T17:56:34Z

Test build #117992 has finished for PR 27477 at commit 4548b0f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-02-06T18:41:44Z

Test build #117994 has finished for PR 27477 at commit 5475675.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-02-06T23:12:50Z

sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4

@@ -748,6 +748,7 @@ predicate
    | NOT? kind=IN '(' expression (',' expression)* ')'
    | NOT? kind=IN '(' query ')'
    | NOT? kind=RLIKE pattern=valueExpression
+    | NOT? kind=LIKE quantifier=(ANY | ALL) '(' expression (',' expression)* ')'


So, we don't need to support ESCAPE. Did I understand correctly?

Yes.

https://docs.teradata.com/reader/756LNiPSFdY~4JcCCcR5Cw/4~AyrPNmDN0Xk4SALLo6aQ

dongjoon-hyun · 2020-02-06T23:16:50Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala

-        }.getOrElse('\\')
-        invertIfNotDefined(Like(e, expression(ctx.pattern), Literal(escapeChar)))
+        Option(ctx.quantifier).map(_.getType) match {
+          case Some(SqlBaseParser.ANY) if !ctx.expression.isEmpty =>


Do we need this ctx.expression.isEmpty? It seems that the parser rule guarantee at least one expression.

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala

sql/core/src/test/resources/sql-tests/inputs/like-any.sql

sql/core/src/test/resources/sql-tests/inputs/like-all.sql

wangyum · 2020-02-07T05:24:00Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala

@@ -1375,6 +1375,14 @@ class AstBuilder(conf: SQLConf) extends SqlBaseBaseVisitor[AnyRef] with Logging
      case other => Seq(other)
    }

+    def getLikeQuantifierExps(expressions: java.util.List[ExpressionContext]): Seq[Expression] = {
+      if (expressions.isEmpty) {
+        throw new ParseException("Syntax error: expected something between '(' and ')'.", ctx)


Teradata's exception:

I think should remove Syntax error: , because ParseException could replace it.

dongjoon-hyun · 2020-02-07T06:25:43Z

sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4

@@ -748,6 +748,7 @@ predicate
    | NOT? kind=IN '(' expression (',' expression)* ')'
    | NOT? kind=IN '(' query ')'
    | NOT? kind=RLIKE pattern=valueExpression
+    | NOT? kind=LIKE quantifier=(ANY | ALL) ('('')' | '(' expression (',' expression)* ')')


What happened previously when we didn't have '('')' | here? I guessed that it was also a Parse Exception.

Otherwise it will throw AnalysisException:

-- !query select company from like_any_table where company like any () -- !query schema struct<> -- !query output org.apache.spark.sql.AnalysisException Invalid number of arguments for function any. Expected: 1; Found: 0; line 1 pos 54

Oh, it's considered as function. I got it.

SparkQA · 2020-02-07T08:05:01Z

Test build #118015 has finished for PR 27477 at commit de7c398.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

sql/core/src/test/resources/sql-tests/inputs/like-all.sql

SparkQA · 2020-02-07T13:24:12Z

Test build #118025 has finished for PR 27477 at commit caea82a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-02-07T18:27:09Z

Test build #118038 has finished for PR 27477 at commit 88ca4c2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

# Conflicts: # sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala

SparkQA · 2020-02-12T19:15:48Z

Test build #118302 has finished for PR 27477 at commit 1c3d98c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2020-02-13T01:52:36Z

cc @cloud-fan

gatorsmile · 2020-02-18T07:56:50Z

This would be good to have since both Teradata and Snowflake support it.

cc @maryannxue @hvanhovell @cloud-fan

SparkQA · 2020-04-11T12:45:09Z

Test build #121127 has finished for PR 27477 at commit 0656b05.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-04-22T13:13:57Z

retest this please

HyukjinKwon · 2020-04-22T13:15:41Z

Looks fine to me

cloud-fan · 2020-04-22T15:33:39Z

sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4

@@ -766,6 +766,7 @@ predicate
    | NOT? kind=IN '(' expression (',' expression)* ')'
    | NOT? kind=IN '(' query ')'
    | NOT? kind=RLIKE pattern=valueExpression
+    | NOT? kind=LIKE quantifier=(ANY | ALL) ('('')' | '(' expression (',' expression)* ')')


shall we support SOME as well? The BoolOr agg func can be called with any and some.

dilipbiswal · 2020-04-22T17:32:14Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala

+            val escapeChar = Option(ctx.escapeChar).map(string).map { str =>
+              if (str.length != 1) {
+                throw new ParseException("Invalid escape string." +
+                  "Escape string must contains only one character.", ctx)


Nit: contains -> contain ?

dilipbiswal · 2020-04-22T17:32:52Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala

+          case Some(SqlBaseParser.ANY) =>
+            getLikeQuantifierExps(ctx.expression).reduceLeft(Or)
+          case Some(SqlBaseParser.ALL) =>
+            getLikeQuantifierExps(ctx.expression).reduceLeft(And)


Nit: getLikeQuantifierExps -> getLikeQuantifierExprs ?

SparkQA · 2020-04-22T19:15:14Z

Test build #121622 has finished for PR 27477 at commit 0656b05.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-04-23T07:56:47Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala

+    assertEqual("not (a like any ('foo%', 'bar%'))", !(('a like "foo%") || ('a like "bar%")))
+    assertEqual("a like all ('foo%', 'bar%')", ('a like "foo%") && ('a like "bar%"))
+    assertEqual("a not like all ('foo%', 'bar%')", !('a like "foo%") && !('a like "bar%"))
+    assertEqual("not (a like all ('foo%', 'bar%'))", !(('a like "foo%") && ('a like "bar%")))


Could you add two more tests for error handling for L1396 and L1422 in AstBuilder.scala?

maropu · 2020-04-23T08:10:23Z

sql/core/src/test/resources/sql-tests/results/like-all.sql.out

+-- Automatically generated by SQLQueryTestSuite
+-- Number of queries: 14
+
+


note: I've checked that the output is the same with PostgreSQL output: https://gist.github.com/maropu/fa4bd6491e21751d6bbc44c545390b0c

maropu · 2020-04-23T08:11:03Z

Looks fine except for the existing comments.

SparkQA · 2020-04-23T18:02:46Z

Test build #121684 has finished for PR 27477 at commit cf4666f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-04-24T07:05:02Z

Test build #121713 has finished for PR 27477 at commit f9af131.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-04-24T07:09:28Z

retest this please

SparkQA · 2020-04-24T12:52:31Z

Test build #121733 has finished for PR 27477 at commit f9af131.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-04-24T13:18:34Z

@wangyum btw, we need to update the SQL document for this new syntax. Follow-up PR is alright, though. cc: @huaxingao

maropu · 2020-04-24T13:21:02Z

Thanks, all! Merged to master.

huaxingao · 2020-04-24T17:15:01Z

@maropu Since this is for 3.1, I will not include this new syntax in 3.0 sql ref.

maropu · 2020-04-24T17:25:38Z

Yea, we need to update it only in master.

MaxGekk · 2021-09-13T08:25:33Z

sql/core/src/test/resources/sql-tests/inputs/like-all.sql

+SELECT company FROM like_all_table WHERE company NOT LIKE ALL (NULL, NULL);
+
+-- negative case
+SELECT company FROM like_any_table WHERE company LIKE ALL ();


Is using of non-existing table intentional? I guess the purpose was to check LIKE ALL ().

I think it's a typo

wangyum added 2 commits February 6, 2020 21:09

Support 'like any' and 'like all' operators

4548b0f

operator -> quantifier

5475675

dongjoon-hyun added the SQL label Feb 6, 2020

dongjoon-hyun reviewed Feb 6, 2020

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala Outdated Show resolved Hide resolved

HyukjinKwon reviewed Feb 7, 2020

View reviewed changes

sql/core/src/test/resources/sql-tests/inputs/like-any.sql Outdated Show resolved Hide resolved

HyukjinKwon reviewed Feb 7, 2020

View reviewed changes

sql/core/src/test/resources/sql-tests/inputs/like-all.sql Outdated Show resolved Hide resolved

address comments

de7c398

wangyum commented Feb 7, 2020

View reviewed changes

dongjoon-hyun reviewed Feb 7, 2020

View reviewed changes

SOME -> ALL

caea82a

maropu reviewed Feb 7, 2020

View reviewed changes

sql/core/src/test/resources/sql-tests/inputs/like-all.sql Outdated Show resolved Hide resolved

Capitalize SQL keywords

88ca4c2

beliefer approved these changes Feb 8, 2020

View reviewed changes

Merge remote-tracking branch 'upstream/master' into SPARK-30724

1c3d98c

# Conflicts: # sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala

wangyum added 2 commits April 11, 2020 13:13

Merge remote-tracking branch 'upstream/master' into SPARK-30724

2b59cb0

Add a test case from our customers

0656b05

cloud-fan reviewed Apr 22, 2020

View reviewed changes

dilipbiswal reviewed Apr 22, 2020

View reviewed changes

maropu reviewed Apr 23, 2020

View reviewed changes

Address comments

cf4666f

cloud-fan approved these changes Apr 23, 2020

View reviewed changes

Fix test error

f9af131

maropu closed this in b10263b Apr 24, 2020

wangyum deleted the SPARK-30724 branch April 24, 2020 13:31

MaxGekk reviewed Sep 13, 2021

View reviewed changes

		-- Automatically generated by SQLQueryTestSuite
		-- Number of queries: 14

[SPARK-30724][SQL] Support 'LIKE ANY' and 'LIKE ALL' operators #27477

[SPARK-30724][SQL] Support 'LIKE ANY' and 'LIKE ALL' operators #27477

Conversation

wangyum commented Feb 6, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Feb 6, 2020

SparkQA commented Feb 6, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Feb 7, 2020

SparkQA commented Feb 7, 2020

SparkQA commented Feb 7, 2020

SparkQA commented Feb 12, 2020

wangyum commented Feb 13, 2020

gatorsmile commented Feb 18, 2020

SparkQA commented Apr 11, 2020

HyukjinKwon commented Apr 22, 2020

HyukjinKwon commented Apr 22, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Apr 22, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maropu commented Apr 23, 2020

SparkQA commented Apr 23, 2020

SparkQA commented Apr 24, 2020

maropu commented Apr 24, 2020

SparkQA commented Apr 24, 2020

maropu commented Apr 24, 2020 • edited Loading

maropu commented Apr 24, 2020

huaxingao commented Apr 24, 2020

maropu commented Apr 24, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangyum commented Feb 6, 2020 •

edited

Loading

maropu commented Apr 24, 2020 •

edited

Loading

maropu commented Apr 24, 2020 •

edited

Loading