Tracking issue for Cardinality Estimation Enhancement

[time-and-fate]

Enhancement

We're investigating flaws in our cardinality estimation logic and trying to fix them. And we're also trying to improve our cardinality estimation mechanism. This issue is for tracking the developments.

• Improve out-of-range handling logic #26086
• Improve row count estimation for Limit/TopN #26088
• Always push down TopN when N is less than a specific variable #26089
• do not convert small range query to multiple point queries under stats-ver2 #26281
• Update the formula about judging stale statistics #26282
• unify the terms NDV and cardinality in the optimizer #26283

[time-and-fate]

/sig planner

[tangwz]

May I have a try?

[time-and-fate]

May I have a try?

Thank you for that.
But some details of these items are still under discussion and we also have to complete them quickly. So I think we probably need to complete them internally.

[tisonkun]

May I have a try?

Thank you for that.
But some details of these items are still under discussion and we also have to complete them quickly. So I think we probably need to complete them internally.

@time-and-fate is there a design document or pre-RFC? We may avoid discuss privately if unnecessary.

[qw4990]

Here are two simple discussion documents about these in this issue: https://docs.google.com/document/d/1IyxlOJAQY-lFpcId0nRJy4aVgm4Be1K5CUESsqSZNWk/edit?usp=sharing, https://docs.google.com/document/d/16sn1aoTfzQXBWR3QAztTpu-JnIzg8r23lMkRT8ua4Ck/edit?usp=sharing.
Since these items are found from our customers' feedbacks, they are relatively urgent and confidential, so we decided to complete them internally.
After finishing these urgent items, we'll publish more improvement items about Cardinality Estimation next sprint. Thanks for your understanding ~~~~ @tisonkun @tangwz

[tisonkun]

@time-and-fate @qw4990 is there any ongoing issues? If all subtasks closed, it should be a conclusion about the status of the tracking issue.

I'd like to ask for a design document or brief description about this "enhancement". Otherwise we implement feature secretly and hard to trace back even by ourselves.

[time-and-fate]

The tasks for this staged have been completed and the design docs are contained in the links above.
We are still planning about the next stage to see what needs to be done and their priority. We'll update this issue if there's any progress.

[tisonkun]

@time-and-fate the document above is private to PingCAP company. I'd prefer to see a brief description public in this PR or as a design document.

[qw4990]

Close since all tasks have been finished.