*: only add default value for final aggregation to fix the aggregate push down (partition) union case #35443

tiancaiamao · 2022-06-16T08:03:26Z

What problem does this PR solve?

Issue Number: close #35295

Problem Summary:

When table t is empty, this get empty result:

select avg(id) from t group by id;

While this get NULL:

select avg(id) from t;

We add NULL (default value) to the empty result here:

tidb/executor/aggregate.go

Lines 1368 to 1379 in 2de01b4

    
           if e.childResult.NumRows() == 0 { 
        
           	if !e.isChildReturnEmpty { 
        
           		err = e.appendResult2Chunk(chk) 
        
           	} else if e.defaultVal != nil { 
        
           		chk.Append(e.defaultVal, 0, 1) 
        
           	} 
        
           	e.executed = true 
        
           	return err 
        
           } 
        
           // Reach here, "e.childrenResults[0].NumRows() > 0" is guaranteed. 
        
           e.isChildReturnEmpty = false 
        
           e.inputRow = e.inputIter.Begin()

Whether or not add the NULL row depends on some conditions.

Note, for this case, the added NULL row make the final result wrong:

What is changed and how it works?

Only add NULL for final aggregation.
Otherwise the add NULL row operation would be done several times and send a wrong NULL row to the final aggregation.

In aggregation push down optimze rule, I'll set the pushed down aggregation to non-final
In physical plan, aggregation push down to cop phase, I'll not blindly change the upper aggregation to final/complete... because it's parent might be final and it's a partial itself
In the executor builder phase, only set the default value for final aggregation

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

…push down (partition) union case

ti-chi-bot · 2022-06-16T08:03:27Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

qw4990
winoros

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

tiancaiamao · 2022-06-17T15:17:57Z

/run-unit-test

tiancaiamao · 2022-06-20T04:47:08Z

/run-unit-test

sre-bot · 2022-06-20T05:32:49Z

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/e799be1077d7efc8c499f0315159a4f81eb06bf7

tiancaiamao · 2022-06-20T05:39:46Z

PTAL @XuHuaiyu @qw4990

XuHuaiyu · 2022-06-20T06:06:04Z

expression/aggregation/descriptor.go

@@ -123,8 +123,6 @@ func (a *AggFuncDesc) Split(ordinal []int) (partialAggDesc, finalAggDesc *AggFun
 		partialAggDesc.Mode = Partial1Mode
 	} else if a.Mode == FinalMode {
 		partialAggDesc.Mode = Partial2Mode
-	} else {


Why remove this check?

In the previous assumption, there will be no partial mode agg before coprocessor push down...

But now I change the logical plan aggregation to partial, so in the coprocessor push down here, this line may meet partial mode agg.

planner/core/physical_plans.go

XuHuaiyu · 2022-06-20T06:21:35Z

planner/core/task.go

@@ -1417,7 +1421,11 @@ func BuildFinalModeAggregation(
 				}
 			}

-			finalAggFunc.Mode = aggregation.FinalMode
+			if aggFunc.Mode == aggregation.CompleteMode {


I do not understand what does this code block mean

In the old assumption, before the coprocessor push down, the agg mode is either Complete or Final. So after the push down, the parent agg become Final mode.

But in the new assumption, before the coprocessor push down, the agg mode can be partial. So here the parent agg mode is set to different value accordingly.

For final before, we set it to final.
For partial, it's set to partial2

winoros · 2022-06-22T16:47:52Z

planner/core/task.go

@@ -1417,7 +1421,11 @@ func BuildFinalModeAggregation(
 				}
 			}

-			finalAggFunc.Mode = aggregation.FinalMode
+			if aggFunc.Mode == aggregation.CompleteMode {


I think you mean that when we are in the partition union mode. The Agg->PartitionUnion->TableReader may become Agg->PartitionUnion->Agg->TableReader, then we do the push down. So we need to add a new if-condition check.

You can add some comments here.

Agg1->PartitionUnion->Agg2->TableReader

In coprocessor push down, Agg2->TableReader become Agg3(root)->TableReader->Agg4(cop)->TableScan(cop), and the final plan become Agg1->PartitionUnion->Agg3(root)->TableReader->Agg4(cop)->TableScan(cop)

In the past, Agg2 is always Complete or Final, but in this PR here Agg2 could be Partial
So BuildFinalModeAggregation need to consider the difference here.

I add some comment in the code so the reviewer know what happen.

Signed-off-by: ti-srebot <[email protected]>

ti-srebot · 2022-06-28T04:09:47Z

cherry pick to release-4.0 in PR #35765

Signed-off-by: ti-srebot <[email protected]>

ti-srebot · 2022-06-28T04:11:06Z

cherry pick to release-5.0 in PR #35766

Signed-off-by: ti-srebot <[email protected]>

ti-srebot · 2022-06-28T04:12:17Z

cherry pick to release-5.1 in PR #35767

Signed-off-by: ti-srebot <[email protected]>

ti-srebot · 2022-06-28T04:13:34Z

cherry pick to release-5.2 in PR #35768

Signed-off-by: ti-srebot <[email protected]>

ti-srebot · 2022-06-28T04:14:42Z

cherry pick to release-5.3 in PR #35769

Signed-off-by: ti-srebot <[email protected]>

ti-srebot · 2022-06-28T04:15:54Z

cherry pick to release-5.4 in PR #35770

Signed-off-by: ti-srebot <[email protected]>

ti-srebot · 2022-06-28T04:17:08Z

cherry pick to release-6.0 in PR #35771

Signed-off-by: ti-srebot <[email protected]>

ti-srebot · 2022-06-28T04:18:27Z

cherry pick to release-6.1 in PR #35772

sre-bot · 2022-06-28T04:41:11Z

TiDB MergeCI notify

🔴 Bad News! [1] CI still failing after this pr merged.
These failed integration tests don't seem to be introduced by the current PR.

CI Name	Result	Duration	Compare with Parent commit
idc-jenkins-ci-tidb/integration-common-test	🔴 failed 2, success 9, total 11	21 min	Existing failure
idc-jenkins-ci/integration-cdc-test	🟢 all 35 tests passed	25 min	Existing passed
idc-jenkins-ci-tidb/common-test	🟢 all 12 tests passed	11 min	Existing passed
idc-jenkins-ci-tidb/integration-ddl-test	🟢 all 6 tests passed	8 min 48 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-2	🟢 all 28 tests passed	7 min 23 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-1	🟢 all 26 tests passed	6 min 54 sec	Existing passed
idc-jenkins-ci-tidb/tics-test	🟢 all 1 tests passed	6 min 7 sec	Existing passed
idc-jenkins-ci-tidb/integration-compatibility-test	🟢 all 1 tests passed	4 min 44 sec	Existing passed
idc-jenkins-ci-tidb/mybatis-test	🟢 all 1 tests passed	3 min 35 sec	Existing passed
idc-jenkins-ci-tidb/plugin-test	🟢 build success, plugin test success	4min	Existing passed

* upstream/master: (57 commits) types: fix incompatible implementation of jsonpath extraction (pingcap#35320) planner: fix TRACE PLAN TARGET = 'estimation' panic when meeting partition table (pingcap#35743) *: Add `testfork.RunTest` to run multiple tests in one function (pingcap#35746) sessionctx/variable: add tests to ensure skipInit can be removed (pingcap#35703) helper: request another PD if one of them is unavailable (pingcap#35750) metrics: add cached table related metrics to grafana panel (pingcap#34718) expression: use cloned RetType at `evaluateExprWithNull` when it may be changed. (pingcap#35759) executor: fix left join on partition table generate invalid lock key (pingcap#35732) readme: remove adopters (pingcap/docs#8725) (pingcap#35124) *: only add default value for final aggregation to fix the aggregate push down (partition) union case (pingcap#35443) planner: fix the wrong cost formula of MPPExchanger on cost model ver2 (pingcap#35718) sessionctx: support encoding and decoding statement context (pingcap#35688) txn: refactor ts acquisition within build and execute phases (pingcap#35376) ddl: for schema-level DDL method parameter is now XXXStmt (pingcap#35722) *: enable gofmt (pingcap#35721) planner: disable collate clause support for enum or set column (pingcap#35684) *: Provide a util to "pause" session in uint test (pingcap#35529) ddl: implement the core for multi-schema change (pingcap#35429) parser: XXXDatabaseStmt now use CIStr for DB name (pingcap#35668) *: remove real tikv test on github actions (pingcap#35710) ...

…push down (partition) union case (#35443) (#35772) close #35295

…push down (partition) union case (#35443) (#35770) close #35295

*: only add default value for final aggregation to fix the aggregate …

86998d1

…push down (partition) union case

tiancaiamao marked this pull request as ready for review June 16, 2022 08:03

ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 16, 2022

tiancaiamao requested review from XuHuaiyu and qw4990 June 16, 2022 08:03

try to fix CI

7d68e32

Merge branch 'master' into issue-35295

7d57765

XuHuaiyu reviewed Jun 20, 2022

View reviewed changes

address comment

b87ea6c

winoros reviewed Jun 22, 2022

View reviewed changes

address comment

ff92581

ti-chi-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 23, 2022

tiancaiamao requested a review from winoros June 23, 2022 03:38

ti-chi-bot merged commit d99b358 into pingcap:master Jun 28, 2022

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Jun 28, 2022

cherry pick pingcap#35443 to release-4.0

40efd0e

Signed-off-by: ti-srebot <[email protected]>

ti-srebot mentioned this pull request Jun 28, 2022

*: only add default value for final aggregation to fix the aggregate push down (partition) union case (#35443) #35765

Closed

12 tasks

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Jun 28, 2022

cherry pick pingcap#35443 to release-5.0

f269269

Signed-off-by: ti-srebot <[email protected]>

ti-srebot mentioned this pull request Jun 28, 2022

*: only add default value for final aggregation to fix the aggregate push down (partition) union case (#35443) #35766

Closed

12 tasks

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Jun 28, 2022

cherry pick pingcap#35443 to release-5.1

2e1f7cc

Signed-off-by: ti-srebot <[email protected]>

ti-srebot mentioned this pull request Jun 28, 2022

*: only add default value for final aggregation to fix the aggregate push down (partition) union case (#35443) #35767

Closed

12 tasks

ti-srebot mentioned this pull request Jun 28, 2022

*: only add default value for final aggregation to fix the aggregate push down (partition) union case (#35443) #35768

Closed

12 tasks

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Jun 28, 2022

cherry pick pingcap#35443 to release-5.2

22dcbdb

Signed-off-by: ti-srebot <[email protected]>

tiancaiamao deleted the issue-35295 branch June 28, 2022 04:14

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Jun 28, 2022

cherry pick pingcap#35443 to release-5.3

8795237

Signed-off-by: ti-srebot <[email protected]>

ti-srebot mentioned this pull request Jun 28, 2022

*: only add default value for final aggregation to fix the aggregate push down (partition) union case (#35443) #35769

Closed

12 tasks

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Jun 28, 2022

cherry pick pingcap#35443 to release-5.4

f3b7f64

Signed-off-by: ti-srebot <[email protected]>

ti-srebot mentioned this pull request Jun 28, 2022

*: only add default value for final aggregation to fix the aggregate push down (partition) union case (#35443) #35770

Merged

12 tasks

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Jun 28, 2022

cherry pick pingcap#35443 to release-6.0

d0f0bbd

Signed-off-by: ti-srebot <[email protected]>

ti-srebot mentioned this pull request Jun 28, 2022

*: only add default value for final aggregation to fix the aggregate push down (partition) union case (#35443) #35771

Closed

12 tasks

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Jun 28, 2022

cherry pick pingcap#35443 to release-6.1

9d04ba5

Signed-off-by: ti-srebot <[email protected]>

ti-srebot mentioned this pull request Jun 28, 2022

*: only add default value for final aggregation to fix the aggregate push down (partition) union case (#35443) #35772

Merged

12 tasks

ti-chi-bot pushed a commit that referenced this pull request Aug 22, 2022

*: only add default value for final aggregation to fix the aggregate …

6d0585b

…push down (partition) union case (#35443) (#35772) close #35295

ti-chi-bot pushed a commit that referenced this pull request Sep 14, 2022

*: only add default value for final aggregation to fix the aggregate …

a6f53b2

…push down (partition) union case (#35443) (#35770) close #35295

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: only add default value for final aggregation to fix the aggregate push down (partition) union case #35443

*: only add default value for final aggregation to fix the aggregate push down (partition) union case #35443

tiancaiamao commented Jun 16, 2022

ti-chi-bot commented Jun 16, 2022 •

edited

Loading

tiancaiamao commented Jun 17, 2022

tiancaiamao commented Jun 20, 2022

sre-bot commented Jun 20, 2022 •

edited

Loading

tiancaiamao commented Jun 20, 2022

XuHuaiyu Jun 20, 2022

tiancaiamao Jun 22, 2022

XuHuaiyu Jun 20, 2022

tiancaiamao Jun 22, 2022

winoros Jun 22, 2022

tiancaiamao Jun 23, 2022 •

edited

Loading

tiancaiamao Jun 23, 2022

ti-srebot commented Jun 28, 2022

ti-srebot commented Jun 28, 2022

ti-srebot commented Jun 28, 2022

ti-srebot commented Jun 28, 2022

ti-srebot commented Jun 28, 2022

ti-srebot commented Jun 28, 2022

ti-srebot commented Jun 28, 2022

ti-srebot commented Jun 28, 2022

sre-bot commented Jun 28, 2022

	if e.childResult.NumRows() == 0 {
	if !e.isChildReturnEmpty {
	err = e.appendResult2Chunk(chk)
	} else if e.defaultVal != nil {
	chk.Append(e.defaultVal, 0, 1)
	}
	e.executed = true
	return err
	}
	// Reach here, "e.childrenResults[0].NumRows() > 0" is guaranteed.
	e.isChildReturnEmpty = false
	e.inputRow = e.inputIter.Begin()

*: only add default value for final aggregation to fix the aggregate push down (partition) union case #35443

*: only add default value for final aggregation to fix the aggregate push down (partition) union case #35443

Conversation

tiancaiamao commented Jun 16, 2022

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Jun 16, 2022 • edited Loading

tiancaiamao commented Jun 17, 2022

tiancaiamao commented Jun 20, 2022

sre-bot commented Jun 20, 2022 • edited Loading

tiancaiamao commented Jun 20, 2022

XuHuaiyu Jun 20, 2022

Choose a reason for hiding this comment

tiancaiamao Jun 22, 2022

Choose a reason for hiding this comment

XuHuaiyu Jun 20, 2022

Choose a reason for hiding this comment

tiancaiamao Jun 22, 2022

Choose a reason for hiding this comment

winoros Jun 22, 2022

Choose a reason for hiding this comment

tiancaiamao Jun 23, 2022 • edited Loading

Choose a reason for hiding this comment

tiancaiamao Jun 23, 2022

Choose a reason for hiding this comment

ti-srebot commented Jun 28, 2022

ti-srebot commented Jun 28, 2022

ti-srebot commented Jun 28, 2022

ti-srebot commented Jun 28, 2022

ti-srebot commented Jun 28, 2022

ti-srebot commented Jun 28, 2022

ti-srebot commented Jun 28, 2022

ti-srebot commented Jun 28, 2022

sre-bot commented Jun 28, 2022

TiDB MergeCI notify

ti-chi-bot commented Jun 16, 2022 •

edited

Loading

sre-bot commented Jun 20, 2022 •

edited

Loading

tiancaiamao Jun 23, 2022 •

edited

Loading