[SPARK-31495][SQL] Support formatted explain for AQE #28271

Ngone51 · 2020-04-20T08:56:30Z

What changes were proposed in this pull request?

To support formatted explain for AQE.

Why are the changes needed?

AQE does not support formatted explain yet. It's good to support it for better user experience, debugging, etc.

Before:

== Physical Plan ==
AdaptiveSparkPlan (1)
+- * HashAggregate (unknown)
   +- CustomShuffleReader (unknown)
      +- ShuffleQueryStage (unknown)
         +- Exchange (unknown)
            +- * HashAggregate (unknown)
               +- * Project (unknown)
                  +- * BroadcastHashJoin Inner BuildRight (unknown)
                     :- * LocalTableScan (unknown)
                     +- BroadcastQueryStage (unknown)
                        +- BroadcastExchange (unknown)
                           +- LocalTableScan (unknown)


(1) AdaptiveSparkPlan
Output [4]: [k#7, count(v1)#32L, sum(v1)#33L, avg(v2)#34]
Arguments: HashAggregate(keys=[k#7], functions=[count(1), sum(cast(v1#8 as bigint)), avg(cast(v2#19 as bigint))]), AdaptiveExecutionContext(org.apache.spark.sql.SparkSession@104ab57b), [PlanAdaptiveSubqueries(Map())], false

After:

== Physical Plan ==
 AdaptiveSparkPlan (14)
 +- * HashAggregate (13)
    +- CustomShuffleReader (12)
       +- ShuffleQueryStage (11)
          +- Exchange (10)
             +- * HashAggregate (9)
                +- * Project (8)
                   +- * BroadcastHashJoin Inner BuildRight (7)
                      :- * Project (2)
                      :  +- * LocalTableScan (1)
                      +- BroadcastQueryStage (6)
                         +- BroadcastExchange (5)
                            +- * Project (4)
                               +- * LocalTableScan (3)


 (1) LocalTableScan [codegen id : 2]
 Output [2]: [_1#x, _2#x]
 Arguments: [_1#x, _2#x]

 (2) Project [codegen id : 2]
 Output [2]: [_1#x AS k#x, _2#x AS v1#x]
 Input [2]: [_1#x, _2#x]

 (3) LocalTableScan [codegen id : 1]
 Output [2]: [_1#x, _2#x]
 Arguments: [_1#x, _2#x]

 (4) Project [codegen id : 1]
 Output [2]: [_1#x AS k#x, _2#x AS v2#x]
 Input [2]: [_1#x, _2#x]

 (5) BroadcastExchange
 Input [2]: [k#x, v2#x]
 Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint))), [id=#x]

 (6) BroadcastQueryStage
 Output [2]: [k#x, v2#x]
 Arguments: 0

 (7) BroadcastHashJoin [codegen id : 2]
 Left keys [1]: [k#x]
 Right keys [1]: [k#x]
 Join condition: None

 (8) Project [codegen id : 2]
 Output [3]: [k#x, v1#x, v2#x]
 Input [4]: [k#x, v1#x, k#x, v2#x]

 (9) HashAggregate [codegen id : 2]
 Input [3]: [k#x, v1#x, v2#x]
 Keys [1]: [k#x]
 Functions [3]: [partial_count(1), partial_sum(cast(v1#x as bigint)), partial_avg(cast(v2#x as bigint))]
 Aggregate Attributes [4]: [count#xL, sum#xL, sum#x, count#xL]
 Results [5]: [k#x, count#xL, sum#xL, sum#x, count#xL]

 (10) Exchange
 Input [5]: [k#x, count#xL, sum#xL, sum#x, count#xL]
 Arguments: hashpartitioning(k#x, 5), true, [id=#x]

 (11) ShuffleQueryStage
 Output [5]: [sum#xL, k#x, sum#x, count#xL, count#xL]
 Arguments: 1

 (12) CustomShuffleReader
 Input [5]: [k#x, count#xL, sum#xL, sum#x, count#xL]
 Arguments: coalesced

 (13) HashAggregate [codegen id : 3]
 Input [5]: [k#x, count#xL, sum#xL, sum#x, count#xL]
 Keys [1]: [k#x]
 Functions [3]: [count(1), sum(cast(v1#x as bigint)), avg(cast(v2#x as bigint))]
 Aggregate Attributes [3]: [count(1)#xL, sum(cast(v1#x as bigint))#xL, avg(cast(v2#x as bigint))#x]
 Results [4]: [k#x, count(1)#xL AS count(v1)#xL, sum(cast(v1#x as bigint))#xL AS sum(v1)#xL, avg(cast(v2#x as bigint))#x AS avg(v2)#x]

 (14) AdaptiveSparkPlan
 Output [4]: [k#x, count(v1)#xL, sum(v1)#xL, avg(v2)#x]
 Arguments: isFinalPlan=true

Does this PR introduce any user-facing change?

No, this should be new feature along with AQE in Spark 3.0.

How was this patch tested?

Added a query file: explain-aqe.sql and a unit test.

sql/core/src/test/scala/org/apache/spark/sql/ExplainSuite.scala

SparkQA · 2020-04-20T14:07:24Z

Test build #121514 has finished for PR 28271 at commit 4808054.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Ngone51 · 2020-04-21T13:22:27Z

retest this please.

SparkQA · 2020-04-21T16:59:02Z

Test build #121585 has finished for PR 28271 at commit 4808054.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-04-21T19:39:33Z

Test build #121590 has finished for PR 28271 at commit 0c7ff4f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-04-22T06:26:39Z

Test build #121600 has finished for PR 28271 at commit 1228d2e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Ngone51 · 2020-04-22T06:28:59Z

cc @cloud-fan @maryannxue

HyukjinKwon · 2020-04-22T08:09:47Z

looks fine to me.

cloud-fan · 2020-04-22T10:43:30Z

sql/core/src/main/scala/org/apache/spark/sql/execution/ExplainUtils.scala

@@ -212,7 +228,7 @@ object ExplainUtils {
              case _ =>
            }
        })
-    }
+      }


wrong indentation.

cloud-fan · 2020-04-22T12:44:05Z

The last commit just changes indentation. Thanks, merging to master/3.0!

### What changes were proposed in this pull request? To support formatted explain for AQE. ### Why are the changes needed? AQE does not support formatted explain yet. It's good to support it for better user experience, debugging, etc. Before: ``` == Physical Plan == AdaptiveSparkPlan (1) +- * HashAggregate (unknown) +- CustomShuffleReader (unknown) +- ShuffleQueryStage (unknown) +- Exchange (unknown) +- * HashAggregate (unknown) +- * Project (unknown) +- * BroadcastHashJoin Inner BuildRight (unknown) :- * LocalTableScan (unknown) +- BroadcastQueryStage (unknown) +- BroadcastExchange (unknown) +- LocalTableScan (unknown) (1) AdaptiveSparkPlan Output [4]: [k#7, count(v1)#32L, sum(v1)#33L, avg(v2)#34] Arguments: HashAggregate(keys=[k#7], functions=[count(1), sum(cast(v1#8 as bigint)), avg(cast(v2#19 as bigint))]), AdaptiveExecutionContext(org.apache.spark.sql.SparkSession104ab57b), [PlanAdaptiveSubqueries(Map())], false ``` After: ``` == Physical Plan == AdaptiveSparkPlan (14) +- * HashAggregate (13) +- CustomShuffleReader (12) +- ShuffleQueryStage (11) +- Exchange (10) +- * HashAggregate (9) +- * Project (8) +- * BroadcastHashJoin Inner BuildRight (7) :- * Project (2) : +- * LocalTableScan (1) +- BroadcastQueryStage (6) +- BroadcastExchange (5) +- * Project (4) +- * LocalTableScan (3) (1) LocalTableScan [codegen id : 2] Output [2]: [_1#x, _2#x] Arguments: [_1#x, _2#x] (2) Project [codegen id : 2] Output [2]: [_1#x AS k#x, _2#x AS v1#x] Input [2]: [_1#x, _2#x] (3) LocalTableScan [codegen id : 1] Output [2]: [_1#x, _2#x] Arguments: [_1#x, _2#x] (4) Project [codegen id : 1] Output [2]: [_1#x AS k#x, _2#x AS v2#x] Input [2]: [_1#x, _2#x] (5) BroadcastExchange Input [2]: [k#x, v2#x] Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint))), [id=#x] (6) BroadcastQueryStage Output [2]: [k#x, v2#x] Arguments: 0 (7) BroadcastHashJoin [codegen id : 2] Left keys [1]: [k#x] Right keys [1]: [k#x] Join condition: None (8) Project [codegen id : 2] Output [3]: [k#x, v1#x, v2#x] Input [4]: [k#x, v1#x, k#x, v2#x] (9) HashAggregate [codegen id : 2] Input [3]: [k#x, v1#x, v2#x] Keys [1]: [k#x] Functions [3]: [partial_count(1), partial_sum(cast(v1#x as bigint)), partial_avg(cast(v2#x as bigint))] Aggregate Attributes [4]: [count#xL, sum#xL, sum#x, count#xL] Results [5]: [k#x, count#xL, sum#xL, sum#x, count#xL] (10) Exchange Input [5]: [k#x, count#xL, sum#xL, sum#x, count#xL] Arguments: hashpartitioning(k#x, 5), true, [id=#x] (11) ShuffleQueryStage Output [5]: [sum#xL, k#x, sum#x, count#xL, count#xL] Arguments: 1 (12) CustomShuffleReader Input [5]: [k#x, count#xL, sum#xL, sum#x, count#xL] Arguments: coalesced (13) HashAggregate [codegen id : 3] Input [5]: [k#x, count#xL, sum#xL, sum#x, count#xL] Keys [1]: [k#x] Functions [3]: [count(1), sum(cast(v1#x as bigint)), avg(cast(v2#x as bigint))] Aggregate Attributes [3]: [count(1)#xL, sum(cast(v1#x as bigint))#xL, avg(cast(v2#x as bigint))#x] Results [4]: [k#x, count(1)#xL AS count(v1)#xL, sum(cast(v1#x as bigint))#xL AS sum(v1)#xL, avg(cast(v2#x as bigint))#x AS avg(v2)#x] (14) AdaptiveSparkPlan Output [4]: [k#x, count(v1)#xL, sum(v1)#xL, avg(v2)#x] Arguments: isFinalPlan=true ``` ### Does this PR introduce any user-facing change? No, this should be new feature along with AQE in Spark 3.0. ### How was this patch tested? Added a query file: `explain-aqe.sql` and a unit test. Closes #28271 from Ngone51/support_formatted_explain_for_aqe. Authored-by: yi.wu <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit 8fbfdb3) Signed-off-by: Wenchen Fan <[email protected]>

Ngone51 · 2020-04-22T14:06:44Z

thanks all!

SparkQA · 2020-04-22T18:03:32Z

Test build #121620 has finished for PR 28271 at commit 5893e40.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-04-23T00:20:23Z

Hi, All.
This seems to break all Jenkins job in branch-3.0. Could you make a follow-up?

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-3.0-test-maven-hadoop-2.7-hive-2.3-jdk-11/278/

 org.scalatest.exceptions.TestFailedException: explain-aqe.sql
Expected "...ondition: None
     [  ]
(9) AdaptiveSparkPl...", but got "...ondition: None
     []
(9) AdaptiveSparkPl..." Result did not match for query #8

Ngone51 · 2020-04-23T01:19:23Z

Sure, thanks!

probot-autolabeler bot added the SQL label Apr 20, 2020

HyukjinKwon reviewed Apr 20, 2020

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/ExplainSuite.scala Show resolved Hide resolved

Ngone51 added 2 commits April 22, 2020 01:13

support aqe formatted explain

532bef8

update

0c7ff4f

Ngone51 force-pushed the support_formatted_explain_for_aqe branch from 4808054 to 0c7ff4f Compare April 21, 2020 17:14

update test

1228d2e

cloud-fan reviewed Apr 22, 2020

View reviewed changes

cloud-fan approved these changes Apr 22, 2020

View reviewed changes

fix indent

5893e40

cloud-fan closed this in 8fbfdb3 Apr 22, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-31495][SQL] Support formatted explain for AQE #28271

[SPARK-31495][SQL] Support formatted explain for AQE #28271

Ngone51 commented Apr 20, 2020 •

edited

Loading

SparkQA commented Apr 20, 2020

Ngone51 commented Apr 21, 2020

SparkQA commented Apr 21, 2020

SparkQA commented Apr 21, 2020

SparkQA commented Apr 22, 2020

Ngone51 commented Apr 22, 2020

HyukjinKwon commented Apr 22, 2020

cloud-fan Apr 22, 2020

cloud-fan commented Apr 22, 2020

Ngone51 commented Apr 22, 2020

SparkQA commented Apr 22, 2020

dongjoon-hyun commented Apr 23, 2020 •

edited

Loading

Ngone51 commented Apr 23, 2020

[SPARK-31495][SQL] Support formatted explain for AQE #28271

[SPARK-31495][SQL] Support formatted explain for AQE #28271

Conversation

Ngone51 commented Apr 20, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Apr 20, 2020

Ngone51 commented Apr 21, 2020

SparkQA commented Apr 21, 2020

SparkQA commented Apr 21, 2020

SparkQA commented Apr 22, 2020

Ngone51 commented Apr 22, 2020

HyukjinKwon commented Apr 22, 2020

cloud-fan Apr 22, 2020

Choose a reason for hiding this comment

cloud-fan commented Apr 22, 2020

Ngone51 commented Apr 22, 2020

SparkQA commented Apr 22, 2020

dongjoon-hyun commented Apr 23, 2020 • edited Loading

Ngone51 commented Apr 23, 2020

Ngone51 commented Apr 20, 2020 •

edited

Loading

dongjoon-hyun commented Apr 23, 2020 •

edited

Loading