Planner: support Join #5320

SeaRise · 2022-07-08T13:57:46Z

What problem does this PR solve?

Issue Number: ref #4739

Problem Summary:

What is changed and how it works?

Add PhysicalJoin
make PhysicalSource.executor_id = child_query_block->root->executor_id and FinalProjection.executor_id = child_physical_plan.executor_id, In order for the physical node to get the executor_id of the actual child.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

ti-chi-bot · 2022-07-08T13:57:49Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

LittleFall
windtalker

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

) ref pingcap#4739

SeaRise · 2022-07-12T17:16:00Z

/run-all-tests

sre-bot · 2022-07-12T17:48:01Z

Coverage for changed files

Filename                                  Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Planner/PhysicalPlan.cpp                       92                27    70.65%          11                 0   100.00%         131                13    90.08%          56                20    64.29%
Planner/PhysicalPlan.h                          1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Planner/PlanType.cpp                           13                 3    76.92%           1                 0   100.00%          29                 6    79.31%          24                 3    87.50%
Planner/PlanType.h                              3                 1    66.67%           3                 1    66.67%           8                 4    50.00%           0                 0         -
Planner/Planner.cpp                            52                13    75.00%           8                 1    87.50%          71                13    81.69%          40                14    65.00%
Planner/plans/PhysicalBinary.h                 76                58    23.68%           4                 1    75.00%          16                 7    56.25%          28                23    17.86%
Planner/plans/PhysicalJoin.cpp                 49                18    63.27%           6                 0   100.00%         141                 5    96.45%          24                10    58.33%
Planner/plans/PhysicalJoin.h                    3                 0   100.00%           3                 0   100.00%           3                 0   100.00%           0                 0         -
Planner/plans/PhysicalProjection.cpp           13                 0   100.00%           6                 0   100.00%          92                 6    93.48%           6                 0   100.00%
Planner/plans/PhysicalSource.cpp               14                 7    50.00%           2                 0   100.00%          11                 0   100.00%           6                 3    50.00%
Planner/plans/PhysicalSource.h                  3                 0   100.00%           3                 0   100.00%           5                 0   100.00%           0                 0         -
Planner/tests/gtest_physical_plan.cpp         254               161    36.61%          11                 0   100.00%         274                 3    98.91%          56                42    25.00%
tests/gtest_planner_interpreter.cpp           230               114    50.43%           7                 0   100.00%         780                 0   100.00%          24                24     0.00%
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                         803               402    49.94%          66                 3    95.45%        1562                57    96.35%         264               139    47.35%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18679      9488             49.20%    211319  95735        54.70%

full coverage report (for internal network access only)

SeaRise · 2022-07-13T01:43:09Z

dbms/src/Flash/tests/gtest_planner_interpreter.cpp

+   Expression: <cast after aggregation>
+    SharedQuery: <restore concurrency>
+     ParallelAggregating, max_threads: 10, final: true
+      Expression x 10: <before aggregation>


projection action to remove useless column.
The output columns of join are {r_a, r_b, join_c, l_a, l_b, join_c}.
The columns agg needed are {r_a, join_c}

Because now PhysicalPlan::outputAndOptimize can work on Join?

yes, after optimize, Useless columns can be destructed earlier.

SeaRise · 2022-07-13T01:44:34Z

dbms/src/Flash/Planner/plans/PhysicalSource.cpp

@@ -27,7 +28,7 @@ PhysicalPlanNodePtr PhysicalSource::build(
    NamesAndTypes schema;
    for (const auto & col : sample_block)
        schema.emplace_back(col.name, col.type);
-    return std::make_shared<PhysicalSource>("source", schema, log->identifier(), sample_block, source_streams);
+    return std::make_shared<PhysicalSource>(executor_id, schema, log->identifier(), sample_block, source_streams);


In order for the physical node to get the executor_id of the actual child

SeaRise · 2022-07-13T02:03:52Z

dbms/src/Flash/Planner/PhysicalPlan.cpp

+        auto left = popBack();
+
+        // use for gtest_physical_plan
+        if (dagContext().isTest() && right->tp() != PlanType::Source)


After DAGQueryBlock removed, it will be removed

SeaRise · 2022-07-14T06:05:09Z

dbms/src/Flash/Planner/Planner.cpp

-        physical_plan.buildSource(input_streams);
+        RUNTIME_ASSERT(!input_streams_vec[i].empty(), log, "input streams cannot be empty");
+        assert(query_block.children[i] && query_block.children[i]->root && query_block.children[i]->root->has_executor_id());
+        physical_plan.buildSource(query_block.children[i]->root->executor_id(), input_streams_vec[i]);


query_block.children[i]->root->executor_id()
In order for the physical node to get the executor_id of the actual child

SeaRise · 2022-07-14T06:06:32Z

dbms/src/Flash/tests/gtest_planner_interpreter.cpp

+   Expression: <cast after aggregation>
+    SharedQuery: <restore concurrency>
+     ParallelAggregating, max_threads: 10, final: true
+      Expression x 10: <before aggregation>


SeaRise · 2022-07-14T06:06:44Z

dbms/src/Flash/tests/gtest_planner_interpreter.cpp

+         Expression: <append join key and join filters for probe side>
+          Expression: <final projection>
+           MockTableScan
+      Expression x 10: <before aggregation>


SeaRise · 2022-07-14T06:06:54Z

dbms/src/Flash/tests/gtest_planner_interpreter.cpp

-               MockExchangeReceiver
-           Expression x 20: <remove useless column after join>
-            NonJoined: <add stream with non_joined_data if full_or_right_join>)";
+           Expression x 20: <before aggregation>


SeaRise · 2022-07-14T06:06:59Z

dbms/src/Flash/tests/gtest_planner_interpreter.cpp

+              Expression: <append join key and join filters for probe side>
+               Expression: <final projection>
+                MockExchangeReceiver
+           Expression x 20: <before aggregation>


SeaRise · 2022-07-15T11:12:08Z

/run-all-tests

sre-bot · 2022-07-15T11:44:54Z

Coverage for changed files

Filename                                  Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Planner/PhysicalPlan.cpp                       92                27    70.65%          11                 0   100.00%         131                13    90.08%          56                20    64.29%
Planner/PhysicalPlan.h                          1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Planner/PlanType.cpp                           13                 3    76.92%           1                 0   100.00%          29                 6    79.31%          24                 3    87.50%
Planner/PlanType.h                              3                 1    66.67%           3                 1    66.67%           8                 4    50.00%           0                 0         -
Planner/Planner.cpp                            52                13    75.00%           8                 1    87.50%          71                13    81.69%          40                14    65.00%
Planner/plans/PhysicalBinary.h                 76                58    23.68%           4                 1    75.00%          16                 7    56.25%          28                23    17.86%
Planner/plans/PhysicalJoin.cpp                 52                18    65.38%           9                 0   100.00%         156                 5    96.79%          24                10    58.33%
Planner/plans/PhysicalJoin.h                    3                 0   100.00%           3                 0   100.00%           3                 0   100.00%           0                 0         -
Planner/plans/PhysicalProjection.cpp           13                 0   100.00%           6                 0   100.00%          92                 6    93.48%           6                 0   100.00%
Planner/plans/PhysicalSource.cpp               14                 7    50.00%           2                 0   100.00%          11                 0   100.00%           6                 3    50.00%
Planner/plans/PhysicalSource.h                  3                 0   100.00%           3                 0   100.00%           5                 0   100.00%           0                 0         -
Planner/tests/gtest_physical_plan.cpp         254               161    36.61%          11                 0   100.00%         274                 3    98.91%          56                42    25.00%
tests/gtest_planner_interpreter.cpp           230               114    50.43%           7                 0   100.00%         780                 0   100.00%          24                24     0.00%
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                         806               402    50.12%          69                 3    95.65%        1577                57    96.39%         264               139    47.35%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18682      9488             49.21%    211334  95757        54.69%

full coverage report (for internal network access only)

SeaRise · 2022-07-18T15:48:36Z

/run-all-tests

sre-bot · 2022-07-18T16:24:13Z

Coverage for changed files

Filename                                  Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Planner/PhysicalPlan.cpp                       92                27    70.65%          11                 0   100.00%         131                13    90.08%          56                20    64.29%
Planner/PhysicalPlan.h                          1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Planner/PlanType.cpp                           13                 3    76.92%           1                 0   100.00%          29                 6    79.31%          24                 3    87.50%
Planner/PlanType.h                              3                 1    66.67%           3                 1    66.67%           8                 4    50.00%           0                 0         -
Planner/Planner.cpp                            53                13    75.47%           9                 1    88.89%          79                13    83.54%          42                14    66.67%
Planner/plans/PhysicalBinary.h                 76                58    23.68%           4                 1    75.00%          16                 7    56.25%          28                23    17.86%
Planner/plans/PhysicalJoin.cpp                 52                18    65.38%           9                 0   100.00%         156                 5    96.79%          24                10    58.33%
Planner/plans/PhysicalJoin.h                    3                 0   100.00%           3                 0   100.00%           3                 0   100.00%           0                 0         -
Planner/plans/PhysicalProjection.cpp           13                 0   100.00%           6                 0   100.00%          92                 6    93.48%           6                 0   100.00%
Planner/plans/PhysicalSource.cpp               14                 7    50.00%           2                 0   100.00%          11                 0   100.00%           6                 3    50.00%
Planner/plans/PhysicalSource.h                  3                 0   100.00%           3                 0   100.00%           5                 0   100.00%           0                 0         -
Planner/tests/gtest_physical_plan.cpp         254               161    36.61%          11                 0   100.00%         274                 3    98.91%          56                42    25.00%
tests/gtest_planner_interpreter.cpp           230               114    50.43%           7                 0   100.00%         780                 0   100.00%          24                24     0.00%
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                         807               402    50.19%          70                 3    95.71%        1585                57    96.40%         266               139    47.74%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18770      9495             49.41%    214447  95786        55.33%

full coverage report (for internal network access only)

windtalker · 2022-07-19T09:01:13Z

dbms/src/Flash/Planner/Planner.cpp

-        RUNTIME_ASSERT(!input_streams.empty(), log, "input streams cannot be empty");
-        physical_plan.buildSource(input_streams);
+        RUNTIME_ASSERT(!input_streams_vec[i].empty(), log, "input streams cannot be empty");
+        assert(query_block.children[i] && query_block.children[i]->root && query_block.children[i]->root->has_executor_id());


It is not true for some list based executors(sent from TiSpark), do we plan to support that in Planner?

for list based executors, input_streams_vec.size() == 0.
So no error will occur here for list based executors :)

do we plan to support that in Planner?
yes

why list based executors has input_streams_vec.size() == 0 ?
does it take another interpreter path?

Because for list based executors , there will only be one query block.
And the source of query block will only be table scan.

for TableScan, the input_streams_vec must be empty

windtalker · 2022-07-19T09:13:22Z

dbms/src/Flash/tests/gtest_planner_interpreter.cpp

+   Expression: <cast after aggregation>
+    SharedQuery: <restore concurrency>
+     ParallelAggregating, max_threads: 10, final: true
+      Expression x 10: <before aggregation>


Because now PhysicalPlan::outputAndOptimize can work on Join?

dbms/src/Flash/Planner/plans/PhysicalJoin.cpp

windtalker

LGTM

LittleFall

others lgtm

dbms/src/Flash/Planner/plans/PhysicalBinary.h

Co-authored-by: Zhi Qi <[email protected]>

SeaRise · 2022-07-25T10:17:21Z

/merge

ti-chi-bot · 2022-07-25T10:17:22Z

@SeaRise: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-07-25T10:17:23Z

This pull request has been accepted and is ready to merge.

Commit hash: a55ff0a

SeaRise · 2022-07-25T10:47:11Z

/merge

ti-chi-bot · 2022-07-25T10:47:12Z

@SeaRise: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-07-25T10:47:14Z

This pull request has been accepted and is ready to merge.

Commit hash: be40431

sre-bot · 2022-07-25T11:24:39Z

Coverage for changed files

Filename                                  Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Planner/PhysicalPlan.cpp                      106                32    69.81%          12                 0   100.00%         160                25    84.38%          68                26    61.76%
Planner/PhysicalPlan.h                          1                 0   100.00%           1                 0   100.00%           1                 0   100.00%           0                 0         -
Planner/PlanType.cpp                           15                 4    73.33%           1                 0   100.00%          33                 8    75.76%          28                 4    85.71%
Planner/PlanType.h                              3                 1    66.67%           3                 1    66.67%           8                 4    50.00%           0                 0         -
Planner/Planner.cpp                            51                 7    86.27%           9                 0   100.00%          81                 2    97.53%          40                 8    80.00%
Planner/plans/PhysicalBinary.h                 76                58    23.68%           4                 1    75.00%          16                 7    56.25%          28                23    17.86%
Planner/plans/PhysicalJoin.cpp                 52                18    65.38%           9                 0   100.00%         159                 5    96.86%          24                10    58.33%
Planner/plans/PhysicalJoin.h                    3                 0   100.00%           3                 0   100.00%           3                 0   100.00%           0                 0         -
Planner/plans/PhysicalProjection.cpp           13                 0   100.00%           6                 0   100.00%          92                 6    93.48%           6                 0   100.00%
Planner/plans/PhysicalSource.cpp               14                 7    50.00%           2                 0   100.00%          11                 0   100.00%           6                 3    50.00%
Planner/plans/PhysicalSource.h                  3                 0   100.00%           3                 0   100.00%           5                 0   100.00%           0                 0         -
Planner/tests/gtest_physical_plan.cpp         276               180    34.78%          12                 0   100.00%         288                 3    98.96%          60                46    23.33%
tests/gtest_planner_interpreter.cpp           230               114    50.43%           7                 0   100.00%         785                 0   100.00%          24                24     0.00%
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                         843               421    50.06%          72                 2    97.22%        1642                60    96.35%         284               144    49.30%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18966      9478             50.03%    216498  95728        55.78%

full coverage report (for internal network access only)

add join

6552178

SeaRise closed this Jul 8, 2022

SeaRise added 3 commits July 12, 2022 17:01

Feature Branch: merge master to planner_refactory branch (pingcap#5353

05fd9bd

) ref pingcap#4739

merge from planner_refactory

897bceb

fix tests

4bc88bf

SeaRise reopened this Jul 12, 2022

SeaRise mentioned this pull request Jul 12, 2022

interpreter planner refactor #4739

Closed

13 tasks

SeaRise changed the title ~~[WIP] Planner: support Join~~ Planner: support Join Jul 13, 2022

ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 13, 2022

SeaRise commented Jul 13, 2022

View reviewed changes

SeaRise requested review from windtalker and LittleFall July 13, 2022 01:45

SeaRise commented Jul 13, 2022

View reviewed changes

SeaRise commented Jul 14, 2022

View reviewed changes

refine

2dc2a34

SeaRise force-pushed the new_support_join branch from cfd6b91 to 2dc2a34 Compare July 15, 2022 11:09

SeaRise force-pushed the planner_refactory branch from 05fd9bd to 7693e14 Compare July 18, 2022 10:14

fix conflict

8e882de

windtalker reviewed Jul 19, 2022

View reviewed changes

SeaRise commented Jul 25, 2022

View reviewed changes

dbms/src/Flash/Planner/plans/PhysicalJoin.cpp Show resolved Hide resolved

SeaRise requested a review from windtalker July 25, 2022 03:48

SeaRise added 3 commits July 25, 2022 14:20

Merge branch 'planner_refactory' into new_support_join

ca71b34

port change from pingcap#5411

ca71b3b

Merge branch 'planner_refactory' into new_support_join

ed054fc

gengliqi self-requested a review July 25, 2022 08:44

windtalker approved these changes Jul 25, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jul 25, 2022

LittleFall approved these changes Jul 25, 2022

View reviewed changes

dbms/src/Flash/Planner/plans/PhysicalBinary.h Outdated Show resolved Hide resolved

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jul 25, 2022

Update dbms/src/Flash/Planner/plans/PhysicalBinary.h

a55ff0a

Co-authored-by: Zhi Qi <[email protected]>

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jul 25, 2022

fix

be40431

ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Jul 25, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jul 25, 2022

ti-chi-bot merged commit a528f48 into pingcap:planner_refactory Jul 25, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Planner: support Join #5320

Planner: support Join #5320

SeaRise commented Jul 8, 2022 •

edited

Loading

ti-chi-bot commented Jul 8, 2022 •

edited

Loading

SeaRise commented Jul 12, 2022

sre-bot commented Jul 12, 2022

SeaRise Jul 13, 2022

windtalker Jul 19, 2022

SeaRise Jul 25, 2022

SeaRise Jul 13, 2022

SeaRise Jul 13, 2022

SeaRise Jul 14, 2022

SeaRise Jul 14, 2022

SeaRise Jul 14, 2022

SeaRise Jul 14, 2022

SeaRise Jul 14, 2022

SeaRise commented Jul 15, 2022

sre-bot commented Jul 15, 2022

SeaRise commented Jul 18, 2022

sre-bot commented Jul 18, 2022

windtalker Jul 19, 2022

SeaRise Jul 25, 2022

SeaRise Jul 25, 2022

LittleFall Jul 25, 2022

SeaRise Jul 25, 2022

SeaRise Jul 25, 2022

LittleFall Jul 25, 2022

windtalker Jul 19, 2022

windtalker left a comment

LittleFall left a comment

SeaRise commented Jul 25, 2022

ti-chi-bot commented Jul 25, 2022

ti-chi-bot commented Jul 25, 2022

SeaRise commented Jul 25, 2022

ti-chi-bot commented Jul 25, 2022

ti-chi-bot commented Jul 25, 2022

sre-bot commented Jul 25, 2022

Planner: support Join #5320

Planner: support Join #5320

Conversation

SeaRise commented Jul 8, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Jul 8, 2022 • edited Loading

SeaRise commented Jul 12, 2022

sre-bot commented Jul 12, 2022

Coverage for changed files

Coverage summary

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SeaRise commented Jul 15, 2022

sre-bot commented Jul 15, 2022

Coverage for changed files

Coverage summary

SeaRise commented Jul 18, 2022

sre-bot commented Jul 18, 2022

Coverage for changed files

Coverage summary

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

windtalker left a comment

Choose a reason for hiding this comment

LittleFall left a comment

Choose a reason for hiding this comment

SeaRise commented Jul 25, 2022

ti-chi-bot commented Jul 25, 2022

ti-chi-bot commented Jul 25, 2022

SeaRise commented Jul 25, 2022

ti-chi-bot commented Jul 25, 2022

ti-chi-bot commented Jul 25, 2022

sre-bot commented Jul 25, 2022

Coverage for changed files

Coverage summary

SeaRise commented Jul 8, 2022 •

edited

Loading

ti-chi-bot commented Jul 8, 2022 •

edited

Loading