[SPARK-33985][SQL][TESTS] Add query test of combine usage of TRANSFORM and CLUSTER BY/ORDER BY #32333
Conversation
…LUSTER BY/ORDER BY
|
ping @maropu @cloud-fan |
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
|
Test build #137915 has finished for PR 32333 at commit
|
|
Test build #137916 has finished for PR 32333 at commit
|
|
@AngersZhuuuu can you retrigger https://github.com/AngersZhuuuu/spark/actions/runs/782439275 please? |
Update the test since old test behavior is not stable |
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
maropu
changed the title
| @@ -342,3 +354,41 @@ SELECT TRANSFORM(b, MAX(a) AS max_a, CAST(sum(c) AS STRING)) | |||
| FROM script_trans | |||
| WHERE a <= 2 | |||
| GROUP BY b; | |||
|
|
|||
| -- SPARK-33985 TRANSFORM with CLUSTER BY/ORDER BY/SORT BY | |||
There was a problem hiding this comment.
nit: SPARK-33985 -> SPARK-33985:
There was a problem hiding this comment.
nit:
SPARK-33985->SPARK-33985:
Done
| @@ -342,3 +354,41 @@ SELECT TRANSFORM(b, MAX(a) AS max_a, CAST(sum(c) AS STRING)) | |||
| FROM script_trans | |||
| WHERE a <= 2 | |||
| GROUP BY b; | |||
|
|
|||
| -- SPARK-33985: TRANSFORM with CLUSTER BY/ORDER BY/SORT BY | |||
| FROM ( | |||
There was a problem hiding this comment.
can we use the standard TRANSFORM syntax? Or we write the tests for hive compatibility?
There was a problem hiding this comment.
can we use the standard TRANSFORM syntax? Or we write the tests for hive compatibility?
Yea, for compatibility, since we have less test about transform before. Since I am always checking transform use case between hive and spark. Since this is a important use case of hive's script transform, so add tests to confirm the compatibility.
There was a problem hiding this comment.
Are MAP and REDUCE really compatible between Spark and Hive?
There was a problem hiding this comment.
Are MAP and REDUCE really compatible between Spark and Hive?
Remove this two case.
|
Test build #137931 has finished for PR 32333 at commit
|
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
|
Test build #137941 has finished for PR 32333 at commit
|
|
Test build #137943 has finished for PR 32333 at commit
|
|
thanks, merging to master! |
What changes were proposed in this pull request?
Under hive's document https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Transform there are many usage about TRANSFORM and CLUSTER BY/ORDER BY, in this pr add some test about this cases.
Why are the changes needed?
Add UT
Does this PR introduce any user-facing change?
No
How was this patch tested?
Added UT