statistics: improve out-of-range estimation strategy

[time-and-fate]

What problem does this PR solve?

Issue Number: close #26086

What is changed and how it works?

• Use a new strategy to estimate out-of-range row count estimation. (In (*Histograms).outOfRangeRowCount())
• Make the out-of-range logic use increaseCount instead of modifyCount.
• Out-of-range check no longer needs to check TopN because now the out-of-range strategy is based on histogram.
• Empty histogram no longer means out-of-range because it's possible that all data is in TopN or all data is NULL.
• IncreaseFactor logic only applies to in-the-range part row count, which means we'll first adjust row count using increaseFactor, then add up the our-of-range part row count. (So you'll see IncreaseFactor logic moved from GetRowCountByXXXRanges into GetColumnRowCount and GetRowCount)
• Fix: when estimate row count for equal condition and using uniform distribution assumption, the NDV should exclude TopN's NDV.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Improve out-of-range row count estimation strategy.

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• qw4990
• rebelice

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[time-and-fate]

/sig planner

[time-and-fate]

/run-check_dev_2

[rebelice]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: fe92bb1

[qw4990]

/merge

[time-and-fate]

/run-check_dev_2

[time-and-fate]

/run-check_dev_2

[zhouqiang-cl]

/merge

[ti-chi-bot]

@zhouqiang-cl: /merge is only allowed for the committers, you can assign this pull request to the committer in list by filling /assign @committer in the comment to help merge this pull request.

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

@time-and-fate: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[lmduean]

"Empty histogram no longer means out-of-range because it's possible that all data is in TopN or all data is NULL."
It doesn't seem reasonable. For example:
create table t3(a int, key a_idx(a));
insert into t3 values (NULL); // repeat 2048
analyze table t3;
insert into t3 values (1);// repeat 1024
explain select count(*) from t3 where a between (1) and (2);

mysql> explain analyze select count(*) from t3 where a between (1) and (2);
+---------------------------+---------+---------+-----------+--------------------------+---------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------+---------+------+
| id                        | estRows | actRows | task      | access object            | execution info                                                                                                                              | operator info                 | memory  | disk |
+---------------------------+---------+---------+-----------+--------------------------+---------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------+---------+------+
| StreamAgg_10              | 1.00    | 1       | root      |                          | time:3.99ms, loops:2                                                                                                                        | funcs:count(1)->Column#3      | 9.49 KB | N/A  |
| └─IndexReader_15          | 0.00    | 1024    | root      |                          | time:3.92ms, loops:2, cop_task: {num: 1, max: 3.81ms, proc_keys: 0, tot_proc: 3ms, rpc_num: 1, rpc_time: 3.8ms, copr_cache_hit_ratio: 0.00} | index:IndexRangeScan_14       | 2.26 KB | N/A  |
|   └─IndexRangeScan_14     | 0.00    | 1024    | cop[tikv] | table:t3, index:a_idx(a) | tikv_task:{time:257µs, loops:1024}                                                                                                          | range:[1,2], keep order:false | N/A     | N/A  |
+---------------------------+---------+---------+-----------+--------------------------+---------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------+---------+------+
3 rows in set (0.00 sec)

@time-and-fate
/cc @time-and-fate

[lmduean]

/assign @time-and-fate