*: fix autoid allocation to avoid quickly exhaustion #15261
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
remove unnecessary methods. "The bigger the interface, the weaker the abstraction -- Rob Pike"
ghost
requested review from
wshwsh12 and
XuHuaiyu
and removed request for
a team
tiancaiamao
requested review from
AilinKid and
Little-Wallace
and removed request for
wshwsh12 and
XuHuaiyu
AilinKid
reviewed
Mar 10, 2020
Codecov Report
@@ Coverage Diff @@
## master #15261 +/- ##
================================================
- Coverage 80.6609% 80.6312% -0.0298%
================================================
Files 503 503
Lines 134981 135285 +304
================================================
+ Hits 108877 109082 +205
- Misses 17705 17714 +9
- Partials 8399 8489 +90 |
|
/run-all-tests |
|
LGTM |
|
PTAL @Little-Wallace |
1 similar comment
|
PTAL @Little-Wallace |
| for i, row := range rows { | ||
| var err error | ||
| if i == 0 { | ||
| _, err = e.addRecordWithAutoIDHint(ctx, row, len(rows)) |
There was a problem hiding this comment.
Do we not need to limit the size of reserveAutoIDCount?
There was a problem hiding this comment.
Unnecessary, the rows count equals to the size of a chunk https://github.com/pingcap/tidb/blob/master/executor/insert_common.go#L420
|
Fix #15073 |
|
/merge |
sre-bot
added
the
status/can-merge
|
/run-all-tests |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
In #13648, we try to allocate continuous row id for a single INSERT statement.
It asserts the rows count changed by a statement is known in advance, obtained through
stmtCtx.RecordRows().However, that's not true.
In
insert into ... from select, we don't know the actual rows count in advance.stmtCtx.RecordRows()is accumulated instmtCtx.AddRecordRowsin each chunk.That means we count the rows repeatedly.
For example,
insert into ... from select100k rows, we assume therowsis known in advance and it's 100k.However, each time the code run to
table.AddRecord, therowsmight beThe total count is much more than 100k, which wastes a lot of auto-ID.
What is changed and how it works?
I rewrite the
addRecordfunction and add a reverse auto-ID hint parameter.The caller would use the hint to guide the
table.AddRecordto preallocate some auto-ID in the statement context.And later use statement cached ones prior to the table's allocator.
Check List
Tests
Code changes