[SPARK-30651][SQL] Add detailed information for Aggregate operators in EXPLAIN FORMATTED #27368

Eric5553 · 2020-01-27T16:34:04Z

What changes were proposed in this pull request?

Currently EXPLAIN FORMATTED only report input attributes of HashAggregate/ObjectHashAggregate/SortAggregate, while EXPLAIN EXTENDED provides more information of Keys, Functions, etc. This PR enhanced EXPLAIN FORMATTED to sync with original explain behavior.

Why are the changes needed?

The newly added EXPLAIN FORMATTED got less information comparing to the original EXPLAIN EXTENDED

Does this PR introduce any user-facing change?

Yes, taking HashAggregate explain result as example.

SQL

EXPLAIN FORMATTED
  SELECT
    COUNT(val) + SUM(key) as TOTAL,
    COUNT(key) FILTER (WHERE val > 1)
  FROM explain_temp1;

EXPLAIN EXTENDED

== Physical Plan ==
*(2) HashAggregate(keys=[], functions=[count(val#6), sum(cast(key#5 as bigint)), count(key#5)], output=[TOTAL#62L, count(key) FILTER (WHERE (val > 1))#71L])
+- Exchange SinglePartition, true, [id=#89]
   +- HashAggregate(keys=[], functions=[partial_count(val#6), partial_sum(cast(key#5 as bigint)), partial_count(key#5) FILTER (WHERE (val#6 > 1))], output=[count#75L, sum#76L, count#77L])
      +- *(1) ColumnarToRow
         +- FileScan parquet default.explain_temp1[key#5,val#6] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/Users/XXX/spark-dev/spark/spark-warehouse/explain_temp1], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<key:int,val:int>

EXPLAIN FORMATTED - BEFORE

== Physical Plan ==
* HashAggregate (5)
+- Exchange (4)
   +- HashAggregate (3)
      +- * ColumnarToRow (2)
         +- Scan parquet default.explain_temp1 (1)

...
...
(5) HashAggregate [codegen id : 2]
Input: [count#91L, sum#92L, count#93L]
...
...

EXPLAIN FORMATTED - AFTER

== Physical Plan ==
* HashAggregate (5)
+- Exchange (4)
   +- HashAggregate (3)
      +- * ColumnarToRow (2)
         +- Scan parquet default.explain_temp1 (1)

...
...
(5) HashAggregate [codegen id : 2]
Input: [count#91L, sum#92L, count#93L]
Keys: []
Functions: [count(val#6), sum(cast(key#5 as bigint)), count(key#5)]
Results: [(count(val#6)#84L + sum(cast(key#5 as bigint))#85L) AS TOTAL#78L, count(key#5)#86L AS count(key) FILTER (WHERE (val > 1))#87L]
Output: [TOTAL#78L, count(key) FILTER (WHERE (val > 1))#87L]
...
...

How was this patch tested?

Three tests added in explain.sql for HashAggregate/ObjectHashAggregate/SortAggregate.

dilipbiswal · 2020-01-27T18:14:38Z

@Eric5553 Thanks for working on this . Looks good to me. cc @cloud-fan

dilipbiswal · 2020-01-27T19:09:18Z

@Eric5553 Since the implementation is same for variations of aggregate operator, i was wondering if it makes sense to have a base class where we put these common code ? what do you think ?

maropu · 2020-01-28T01:04:41Z

ok to test

Eric5553 · 2020-01-28T02:11:51Z

@Eric5553 Since the implementation is same for variations of aggregate operator, i was wondering if it makes sense to have a base class where we put these common code ? what do you think ?

@dilipbiswal Thanks so much for review! Yeah, this is a concern when I implement for the three aggregate operators. The groupingExpressions(shown as 'Keys') and aggregateExpressions(shown as 'Functions') are defined in each aggregate operator but not in common super class. So I think we cannot abstract the verboseStringWithOperatorId logic here until we abstract these aggregate attributes.

I think the visitor pattern proposed in the discussion of your initial PR would provide more flexibility. By then, we could separate input/output as a common rule for example.

I can give a try if you got any suggestion on this concern, thanks!

maropu · 2020-01-28T04:09:43Z

You cannot do it like this?


abstract class XXXX extends UnaryExecNode  {
  def groupingExpressions: Seq[NamedExpression]
  def aggregateExpressions: Seq[AggregateExpression]
  ...
}

case class HashAggregateExec(
    requiredChildDistributionExpressions: Option[Seq[Expression]],
    groupingExpressions: Seq[NamedExpression],
    aggregateExpressions: Seq[AggregateExpression],
    ...)
  extends XXXX with BlockingOperatorWithCodegen with AliasAwareOutputPartitioning {
  ...

SparkQA · 2020-01-28T05:25:39Z

Test build #117455 has finished for PR 27368 at commit 44a84d2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Eric5553 · 2020-01-28T06:45:01Z

@maropu Sure, I'll try with it. Thanks!

Eric5553 · 2020-01-28T07:51:54Z

@maropu @dilipbiswal I've abstracted the EXPLAIN FORMATTED logic in c5946a3c1c41341a88df2101bbfe44385d3f5c37, please help review. And do we need to filter more common logic for the three aggregate operators? Thanks!

SparkQA · 2020-01-28T08:05:02Z

Test build #117468 has finished for PR 27368 at commit c5946a3.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class AggregateExec(

Eric5553 · 2020-01-28T08:28:48Z

retest this please

SparkQA · 2020-01-28T12:52:48Z

Test build #117477 has finished for PR 27368 at commit 9fabb05.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class AggregateExec(

dilipbiswal · 2020-01-29T02:13:11Z

And do we need to filter more common logic for the three aggregate operators

I think it's a good idea. What do you think @maropu

About the changes, in the cases where Keys or Functions are empty, does it make sense
to not print them ? cc @maropu @cloud-fan for their opinion.

sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggregateExec.scala

Eric5553 · 2020-02-04T13:05:13Z

About the changes, in the cases where Keys or Functions are empty, does it make sense
to not print them ? cc @maropu @cloud-fan for their opinion.

I think we can keep empty Keys or Functions printed, which means the node has no Keys or Functions. Otherwise we don't know if the explain message means no Keys/Functions or we missed the details for them.
What do you think? @dilipbiswal @maropu @cloud-fan

cloud-fan · 2020-02-04T13:12:22Z

makes sense to me

SparkQA · 2020-02-04T16:47:28Z

Test build #117837 has finished for PR 27368 at commit 9c4fc24.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-02-05T05:08:33Z

I think we can keep empty Keys or Functions printed, which means the node has no Keys or Functions. Otherwise we don't know if the explain message means no Keys/Functions or we missed the details for them.
What do you think? @dilipbiswal @maropu @cloud-fan

+1, too.

And do we need to filter more common logic for the three aggregate operators
I think it's a good idea. What do you think @maropu

Looks fine to me, but can you address it in follow-up?

sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggregateExec.scala

maropu · 2020-02-05T05:13:22Z

I left some minor comments and the other parts looks fine to me.

Eric5553 · 2020-02-05T06:34:13Z

I left some minor comments and the other parts looks fine to me.

I've addressed them in ec029df372bccf11ece3349b35cfa87232886505. Thanks so much for the review!

SparkQA · 2020-02-05T08:05:02Z

Test build #117900 has finished for PR 27368 at commit ec029df.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class BaseAggregateExec extends UnaryExecNode

Eric5553 · 2020-02-05T08:42:11Z

retest this please

cloud-fan · 2020-02-05T10:30:30Z

retest this please

SparkQA · 2020-02-05T15:35:10Z

Test build #117925 has finished for PR 27368 at commit ec029df.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class BaseAggregateExec extends UnaryExecNode

SparkQA · 2020-02-05T20:37:09Z

Test build #117939 has finished for PR 27368 at commit cd3b444.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class BaseAggregateExec extends UnaryExecNode

SparkQA · 2020-02-08T14:41:26Z

Test build #118068 has finished for PR 27368 at commit 5b91b19.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2020-02-10T04:33:46Z

Also cc @maryannxue @hvanhovell

cloud-fan · 2020-02-10T08:43:39Z

sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/BaseAggregateExec.scala

+  val groupingExpressions: Seq[NamedExpression]
+  val aggregateExpressions: Seq[AggregateExpression]
+  val aggregateAttributes: Seq[Attribute]
+  val resultExpressions: Seq[NamedExpression]


These can be def, then we don't need to add override val in the aggregate classes.

@cloud-fan @HyukjinKwon Thanks for review, updated to def in dd0988a.

sql/core/src/test/resources/sql-tests/results/explain.sql.out

HyukjinKwon · 2020-02-10T09:42:05Z

sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/BaseAggregateExec.scala

+/**
+ * Holds common logic for aggregate operators
+ */
+abstract class BaseAggregateExec extends UnaryExecNode {


Shall we make it trait?

I see, changed to trait to make it consistent with other operators, e.g. HashJoin BaseLimitExec.

SparkQA · 2020-02-10T13:50:07Z

Test build #118154 has finished for PR 27368 at commit dd0988a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait BaseAggregateExec extends UnaryExecNode

Eric5553 · 2020-02-10T15:47:20Z

retest this please

SparkQA · 2020-02-10T20:39:29Z

Test build #118171 has finished for PR 27368 at commit dd0988a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait BaseAggregateExec extends UnaryExecNode

cloud-fan · 2020-02-12T18:00:30Z

EXPLAIN FORMATTED is a new feature in 3.0 and this is a followup, so I'm merging to 3.0 as well.

Thanks, merging to master/3.0!

…n EXPLAIN FORMATTED ### What changes were proposed in this pull request? Currently `EXPLAIN FORMATTED` only report input attributes of HashAggregate/ObjectHashAggregate/SortAggregate, while `EXPLAIN EXTENDED` provides more information of Keys, Functions, etc. This PR enhanced `EXPLAIN FORMATTED` to sync with original explain behavior. ### Why are the changes needed? The newly added `EXPLAIN FORMATTED` got less information comparing to the original `EXPLAIN EXTENDED` ### Does this PR introduce any user-facing change? Yes, taking HashAggregate explain result as example. **SQL** ``` EXPLAIN FORMATTED SELECT COUNT(val) + SUM(key) as TOTAL, COUNT(key) FILTER (WHERE val > 1) FROM explain_temp1; ``` **EXPLAIN EXTENDED** ``` == Physical Plan == *(2) HashAggregate(keys=[], functions=[count(val#6), sum(cast(key#5 as bigint)), count(key#5)], output=[TOTAL#62L, count(key) FILTER (WHERE (val > 1))#71L]) +- Exchange SinglePartition, true, [id=#89] +- HashAggregate(keys=[], functions=[partial_count(val#6), partial_sum(cast(key#5 as bigint)), partial_count(key#5) FILTER (WHERE (val#6 > 1))], output=[count#75L, sum#76L, count#77L]) +- *(1) ColumnarToRow +- FileScan parquet default.explain_temp1[key#5,val#6] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/Users/XXX/spark-dev/spark/spark-warehouse/explain_temp1], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<key:int,val:int> ``` **EXPLAIN FORMATTED - BEFORE** ``` == Physical Plan == * HashAggregate (5) +- Exchange (4) +- HashAggregate (3) +- * ColumnarToRow (2) +- Scan parquet default.explain_temp1 (1) ... ... (5) HashAggregate [codegen id : 2] Input: [count#91L, sum#92L, count#93L] ... ... ``` **EXPLAIN FORMATTED - AFTER** ``` == Physical Plan == * HashAggregate (5) +- Exchange (4) +- HashAggregate (3) +- * ColumnarToRow (2) +- Scan parquet default.explain_temp1 (1) ... ... (5) HashAggregate [codegen id : 2] Input: [count#91L, sum#92L, count#93L] Keys: [] Functions: [count(val#6), sum(cast(key#5 as bigint)), count(key#5)] Results: [(count(val#6)#84L + sum(cast(key#5 as bigint))#85L) AS TOTAL#78L, count(key#5)#86L AS count(key) FILTER (WHERE (val > 1))#87L] Output: [TOTAL#78L, count(key) FILTER (WHERE (val > 1))#87L] ... ... ``` ### How was this patch tested? Three tests added in explain.sql for HashAggregate/ObjectHashAggregate/SortAggregate. Closes #27368 from Eric5553/ExplainFormattedAgg. Authored-by: Eric Wu <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit 5919bd3) Signed-off-by: Wenchen Fan <[email protected]>

Eric5553 · 2020-02-13T01:54:43Z

@gatorsmile @cloud-fan @dilipbiswal @maropu @HyukjinKwon , thanks so much for your help!

### What changes were proposed in this pull request? The style of `EXPLAIN FORMATTED` output needs to be improved. We’ve already got some observations/ideas in #27368 (comment) #27368 (comment) Observations/Ideas: 1. Using comma as the separator is not clear, especially commas are used inside the expressions too. 2. Show the column counts first? For example, `Results [4]: …` 3. Currently the attribute names are automatically generated, this need to refined. 4. Add arguments field in common implementations as `EXPLAIN EXTENDED` did by calling `argString` in `TreeNode.simpleString`. This will eliminate most existing minor differences between `EXPLAIN EXTENDED` and `EXPLAIN FORMATTED`. 5. Another improvement we can do is: the generated alias shouldn't include attribute id. collect_set(val, 0, 0)#123 looks clearer than collect_set(val#456, 0, 0)#123 This PR is currently addressing comments 2 & 4, and open for more discussions on improving readability. ### Why are the changes needed? The readability of `EXPLAIN FORMATTED` need to be improved, which will help user better understand the query plan. ### Does this PR introduce any user-facing change? Yes, `EXPLAIN FORMATTED` output style changed. ### How was this patch tested? Update expect results of test cases in explain.sql Closes #27509 from Eric5553/ExplainFormattedRefine. Authored-by: Eric Wu <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

### What changes were proposed in this pull request? The style of `EXPLAIN FORMATTED` output needs to be improved. We’ve already got some observations/ideas in #27368 (comment) #27368 (comment) Observations/Ideas: 1. Using comma as the separator is not clear, especially commas are used inside the expressions too. 2. Show the column counts first? For example, `Results [4]: …` 3. Currently the attribute names are automatically generated, this need to refined. 4. Add arguments field in common implementations as `EXPLAIN EXTENDED` did by calling `argString` in `TreeNode.simpleString`. This will eliminate most existing minor differences between `EXPLAIN EXTENDED` and `EXPLAIN FORMATTED`. 5. Another improvement we can do is: the generated alias shouldn't include attribute id. collect_set(val, 0, 0)#123 looks clearer than collect_set(val#456, 0, 0)#123 This PR is currently addressing comments 2 & 4, and open for more discussions on improving readability. ### Why are the changes needed? The readability of `EXPLAIN FORMATTED` need to be improved, which will help user better understand the query plan. ### Does this PR introduce any user-facing change? Yes, `EXPLAIN FORMATTED` output style changed. ### How was this patch tested? Update expect results of test cases in explain.sql Closes #27509 from Eric5553/ExplainFormattedRefine. Authored-by: Eric Wu <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit 1f0300f) Signed-off-by: Wenchen Fan <[email protected]>

…n EXPLAIN FORMATTED ### What changes were proposed in this pull request? Currently `EXPLAIN FORMATTED` only report input attributes of HashAggregate/ObjectHashAggregate/SortAggregate, while `EXPLAIN EXTENDED` provides more information of Keys, Functions, etc. This PR enhanced `EXPLAIN FORMATTED` to sync with original explain behavior. ### Why are the changes needed? The newly added `EXPLAIN FORMATTED` got less information comparing to the original `EXPLAIN EXTENDED` ### Does this PR introduce any user-facing change? Yes, taking HashAggregate explain result as example. **SQL** ``` EXPLAIN FORMATTED SELECT COUNT(val) + SUM(key) as TOTAL, COUNT(key) FILTER (WHERE val > 1) FROM explain_temp1; ``` **EXPLAIN EXTENDED** ``` == Physical Plan == *(2) HashAggregate(keys=[], functions=[count(val#6), sum(cast(key#5 as bigint)), count(key#5)], output=[TOTAL#62L, count(key) FILTER (WHERE (val > 1))#71L]) +- Exchange SinglePartition, true, [id=apache#89] +- HashAggregate(keys=[], functions=[partial_count(val#6), partial_sum(cast(key#5 as bigint)), partial_count(key#5) FILTER (WHERE (val#6 > 1))], output=[count#75L, sum#76L, count#77L]) +- *(1) ColumnarToRow +- FileScan parquet default.explain_temp1[key#5,val#6] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/Users/XXX/spark-dev/spark/spark-warehouse/explain_temp1], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<key:int,val:int> ``` **EXPLAIN FORMATTED - BEFORE** ``` == Physical Plan == * HashAggregate (5) +- Exchange (4) +- HashAggregate (3) +- * ColumnarToRow (2) +- Scan parquet default.explain_temp1 (1) ... ... (5) HashAggregate [codegen id : 2] Input: [count#91L, sum#92L, count#93L] ... ... ``` **EXPLAIN FORMATTED - AFTER** ``` == Physical Plan == * HashAggregate (5) +- Exchange (4) +- HashAggregate (3) +- * ColumnarToRow (2) +- Scan parquet default.explain_temp1 (1) ... ... (5) HashAggregate [codegen id : 2] Input: [count#91L, sum#92L, count#93L] Keys: [] Functions: [count(val#6), sum(cast(key#5 as bigint)), count(key#5)] Results: [(count(val#6)#84L + sum(cast(key#5 as bigint))#85L) AS TOTAL#78L, count(key#5)#86L AS count(key) FILTER (WHERE (val > 1))#87L] Output: [TOTAL#78L, count(key) FILTER (WHERE (val > 1))#87L] ... ... ``` ### How was this patch tested? Three tests added in explain.sql for HashAggregate/ObjectHashAggregate/SortAggregate. Closes apache#27368 from Eric5553/ExplainFormattedAgg. Authored-by: Eric Wu <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

### What changes were proposed in this pull request? The style of `EXPLAIN FORMATTED` output needs to be improved. We’ve already got some observations/ideas in apache#27368 (comment) apache#27368 (comment) Observations/Ideas: 1. Using comma as the separator is not clear, especially commas are used inside the expressions too. 2. Show the column counts first? For example, `Results [4]: …` 3. Currently the attribute names are automatically generated, this need to refined. 4. Add arguments field in common implementations as `EXPLAIN EXTENDED` did by calling `argString` in `TreeNode.simpleString`. This will eliminate most existing minor differences between `EXPLAIN EXTENDED` and `EXPLAIN FORMATTED`. 5. Another improvement we can do is: the generated alias shouldn't include attribute id. collect_set(val, 0, 0)apache#123 looks clearer than collect_set(val#456, 0, 0)apache#123 This PR is currently addressing comments 2 & 4, and open for more discussions on improving readability. ### Why are the changes needed? The readability of `EXPLAIN FORMATTED` need to be improved, which will help user better understand the query plan. ### Does this PR introduce any user-facing change? Yes, `EXPLAIN FORMATTED` output style changed. ### How was this patch tested? Update expect results of test cases in explain.sql Closes apache#27509 from Eric5553/ExplainFormattedRefine. Authored-by: Eric Wu <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

…n EXPLAIN FORMATTED Currently `EXPLAIN FORMATTED` only report input attributes of HashAggregate/ObjectHashAggregate/SortAggregate, while `EXPLAIN EXTENDED` provides more information of Keys, Functions, etc. This PR enhanced `EXPLAIN FORMATTED` to sync with original explain behavior. The newly added `EXPLAIN FORMATTED` got less information comparing to the original `EXPLAIN EXTENDED` Yes, taking HashAggregate explain result as example. **SQL** ``` EXPLAIN FORMATTED SELECT COUNT(val) + SUM(key) as TOTAL, COUNT(key) FILTER (WHERE val > 1) FROM explain_temp1; ``` **EXPLAIN EXTENDED** ``` == Physical Plan == *(2) HashAggregate(keys=[], functions=[count(val#6), sum(cast(key#5 as bigint)), count(key#5)], output=[TOTAL#62L, count(key) FILTER (WHERE (val > 1))#71L]) +- Exchange SinglePartition, true, [id=#89] +- HashAggregate(keys=[], functions=[partial_count(val#6), partial_sum(cast(key#5 as bigint)), partial_count(key#5) FILTER (WHERE (val#6 > 1))], output=[count#75L, sum#76L, count#77L]) +- *(1) ColumnarToRow +- FileScan parquet default.explain_temp1[key#5,val#6] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/Users/XXX/spark-dev/spark/spark-warehouse/explain_temp1], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<key:int,val:int> ``` **EXPLAIN FORMATTED - BEFORE** ``` == Physical Plan == * HashAggregate (5) +- Exchange (4) +- HashAggregate (3) +- * ColumnarToRow (2) +- Scan parquet default.explain_temp1 (1) ... ... (5) HashAggregate [codegen id : 2] Input: [count#91L, sum#92L, count#93L] ... ... ``` **EXPLAIN FORMATTED - AFTER** ``` == Physical Plan == * HashAggregate (5) +- Exchange (4) +- HashAggregate (3) +- * ColumnarToRow (2) +- Scan parquet default.explain_temp1 (1) ... ... (5) HashAggregate [codegen id : 2] Input: [count#91L, sum#92L, count#93L] Keys: [] Functions: [count(val#6), sum(cast(key#5 as bigint)), count(key#5)] Results: [(count(val#6)#84L + sum(cast(key#5 as bigint))#85L) AS TOTAL#78L, count(key#5)#86L AS count(key) FILTER (WHERE (val > 1))#87L] Output: [TOTAL#78L, count(key) FILTER (WHERE (val > 1))#87L] ... ... ``` Three tests added in explain.sql for HashAggregate/ObjectHashAggregate/SortAggregate. Closes apache#27368 from Eric5553/ExplainFormattedAgg. Authored-by: Eric Wu <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

Eric5553 force-pushed the ExplainFormattedAgg branch from c5946a3 to 9fabb05 Compare January 28, 2020 08:49

cloud-fan reviewed Feb 3, 2020

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggregateExec.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Feb 3, 2020

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggregateExec.scala Outdated Show resolved Hide resolved

maropu reviewed Feb 5, 2020

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggregateExec.scala Outdated Show resolved Hide resolved

maropu reviewed Feb 5, 2020

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggregateExec.scala Outdated Show resolved Hide resolved

Eric5553 force-pushed the ExplainFormattedAgg branch from ec029df to cd3b444 Compare February 5, 2020 15:45

dongjoon-hyun added the SQL label Feb 5, 2020

Eric5553 added 3 commits February 8, 2020 18:46

Add function buffer attributes

5e5d481

Shwo aggregation attributes

70cb2df

Add override to improve readability

5b91b19

Eric5553 force-pushed the ExplainFormattedAgg branch from 3a2f7dc to 5b91b19 Compare February 8, 2020 10:46

This was referenced Feb 9, 2020

[SPARK-30764][SQL] Improve the readability of EXPLAIN FORMATTED style #27509

Closed

[SPARK-30765][SQL] Refine base operator abstraction code style #27511

Closed

cloud-fan reviewed Feb 10, 2020

View reviewed changes

sql/core/src/test/resources/sql-tests/results/explain.sql.out Show resolved Hide resolved

cloud-fan approved these changes Feb 10, 2020

View reviewed changes

HyukjinKwon reviewed Feb 10, 2020

View reviewed changes

Address comments of abstraction

dd0988a

cloud-fan closed this in 5919bd3 Feb 12, 2020

Eric5553 mentioned this pull request Feb 24, 2020

[SPARK-30940][SQL] Remove attributeId in auto-generated arguments when Explain SQL query #27685

Closed

Eric5553 deleted the ExplainFormattedAgg branch March 13, 2020 06:50

Nnicolini mentioned this pull request Jun 11, 2020

Nn/spark 31620 palantir/spark#690

Closed

[SPARK-30651][SQL] Add detailed information for Aggregate operators in EXPLAIN FORMATTED #27368

[SPARK-30651][SQL] Add detailed information for Aggregate operators in EXPLAIN FORMATTED #27368

Conversation

Eric5553 commented Jan 27, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

dilipbiswal commented Jan 27, 2020 • edited Loading

dilipbiswal commented Jan 27, 2020

maropu commented Jan 28, 2020

Eric5553 commented Jan 28, 2020 • edited Loading

maropu commented Jan 28, 2020

SparkQA commented Jan 28, 2020

Eric5553 commented Jan 28, 2020

Eric5553 commented Jan 28, 2020

SparkQA commented Jan 28, 2020

Eric5553 commented Jan 28, 2020

SparkQA commented Jan 28, 2020

dilipbiswal commented Jan 29, 2020

Eric5553 commented Feb 4, 2020 • edited Loading

cloud-fan commented Feb 4, 2020

SparkQA commented Feb 4, 2020

maropu commented Feb 5, 2020 • edited Loading

maropu commented Feb 5, 2020

Eric5553 commented Feb 5, 2020

SparkQA commented Feb 5, 2020

Eric5553 commented Feb 5, 2020

cloud-fan commented Feb 5, 2020

SparkQA commented Feb 5, 2020

SparkQA commented Feb 5, 2020

SparkQA commented Feb 8, 2020

gatorsmile commented Feb 10, 2020

cloud-fan Feb 10, 2020

Choose a reason for hiding this comment

HyukjinKwon Feb 10, 2020

Choose a reason for hiding this comment

Eric5553 Feb 10, 2020

Choose a reason for hiding this comment

HyukjinKwon Feb 10, 2020

Choose a reason for hiding this comment

Eric5553 Feb 10, 2020

Choose a reason for hiding this comment

SparkQA commented Feb 10, 2020

Eric5553 commented Feb 10, 2020

SparkQA commented Feb 10, 2020

cloud-fan commented Feb 12, 2020

Eric5553 commented Feb 13, 2020

Eric5553 commented Jan 27, 2020 •

edited

Loading

dilipbiswal commented Jan 27, 2020 •

edited

Loading

Eric5553 commented Jan 28, 2020 •

edited

Loading

Eric5553 commented Feb 4, 2020 •

edited

Loading

maropu commented Feb 5, 2020 •

edited

Loading