Make performance of TPCH q15 stable (#4570)

[ti-chi-bot]

This is an automated cherry-pick of #4570

Signed-off-by: xufei [email protected]

What problem does this PR solve?

Issue Number: close #4451

Problem Summary:

What is changed and how it works?

Actually, there are 2 possible solutions

• Consider local delta memory usage when estimate the result_size_bytes in aggregator on each execute of executeOnBlock
• In Aggregator::mergeAndConvertToBlocks, check the result_size_bytes, and if it exceeds the threshold, convert all the hash table into two-level hash table

For the first solution, converting hash table to two-level hash table can be done by each threads in the first stage of ParallelAggregating, and for the second solution, this converting things are executed in 1 threads.

I've done some test for both solutions, and found first solution has ~20% perfomance gain compared to the second solution for TPCH q15. So I choose the first solution.

Why not enable two-level hash table by default for all the cases:

• not all the agg method support two-level hash table
• test results shows that always use two-level hash table is slower if the table data is small, and the size of agg results is small.

Testing query:
select count(*),id from test group by id;
test has 65536 rows, all rows have the same id, that is to say the above query return only 1 row
Testing cluster has only 1 TiFlash, and the cpu has 36 core, each query is running 1000 times.

	currency = 1	currency = 5
two level hash table	22s	40s
normal hash table	16s	26s

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• fzhedu
• windtalker

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[windtalker]

LGTM

[fzhedu]

LGTM

[windtalker]

/merge

[ti-chi-bot]

@windtalker: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 112bc2b

[windtalker]

/merge

[ti-chi-bot]

@windtalker: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 3336dc4

[sre-bot]

lines: 36.0% (42939 out of 119384)
branches: 5.4% (65797 out of 1221606)