Support txn file transaction #1119
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Evan Zhou <[email protected]>
Signed-off-by: Ping Yu <[email protected]>
Signed-off-by: Ping Yu <[email protected]>
Signed-off-by: Ping Yu <[email protected]>
…1285) * *: update pd client to latest master (#1283) Signed-off-by: disksing <[email protected]> * fix go.mod Signed-off-by: Ping Yu <[email protected]> * loop of LocateKey to get overlapping regions Signed-off-by: Ping Yu <[email protected]> * polish Signed-off-by: Ping Yu <[email protected]> --------- Signed-off-by: disksing <[email protected]> Signed-off-by: Ping Yu <[email protected]> Co-authored-by: disksing <[email protected]>
* pick primary batch by primary key Signed-off-by: Ping Yu <[email protected]> * sort batches by region Signed-off-by: Ping Yu <[email protected]> --------- Signed-off-by: Ping Yu <[email protected]>
* remove retry Signed-off-by: Ping Yu <[email protected]> * polish Signed-off-by: Ping Yu <[email protected]> * revert ignore pre split error Signed-off-by: Ping Yu <[email protected]> --------- Signed-off-by: Ping Yu <[email protected]>
Signed-off-by: Ping Yu <[email protected]>
Signed-off-by: Ping Yu <[email protected]>
… file (#1327) Signed-off-by: Ping Yu <[email protected]>
Signed-off-by: Ping Yu <[email protected]>
Signed-off-by: Ping Yu <[email protected]>
pingyu
changed the title
ystaticy
reviewed
May 7, 2024
| "github.com/stretchr/testify/require" | ||
| ) | ||
|
|
||
| func TestMergeRangesSearchOverlapped(t *testing.T) { |
There was a problem hiding this comment.
How about add some comments to the test case？
ystaticy
reviewed
May 7, 2024
kv/variables.go
Outdated
| @@ -57,6 +60,7 @@ func NewVariables(killed *uint32) *Variables { | |||
| BackoffLockFast: DefBackoffLockFast, | |||
| BackOffWeight: DefBackOffWeight, | |||
| Killed: killed, | |||
| EnableTxnFile: true, | |||
There was a problem hiding this comment.
it’s better to use GLOBAL variables to control whether txn file transaction is enabled or not.
It allows user to try it out on some clusters first. We have serverless version in TiDB which is very convenient to turn on all switches.
You can see serverless version in
https://github.com/tidbcloud/tidb-cse/blob/release-7.1-keyspace/session/bootstrap_serverless.go
There was a problem hiding this comment.
Yes it's designed to be used for variables. And here is the client-go part.
The tidb-cse part will be added after here:
{Scope: ScopeGlobal | ScopeSession, Name: TiDBBackoffLockFast, Value: strconv.Itoa(tikvstore.DefBackoffLockFast), Type: TypeUnsigned, MinValue: 1, MaxValue: math.MaxInt32, SetSession: func(s *SessionVars, val string) error {
s.KVVars.BackoffLockFast = tidbOptPositiveInt32(val, tikvstore.DefBackoffLockFast)
return nil
}},
{Scope: ScopeGlobal | ScopeSession, Name: TiDBBackOffWeight, Value: strconv.Itoa(tikvstore.DefBackOffWeight), Type: TypeUnsigned, MinValue: 0, MaxValue: math.MaxInt32, SetSession: func(s *SessionVars, val string) error {
s.KVVars.BackOffWeight = tidbOptPositiveInt32(val, tikvstore.DefBackOffWeight)
return nil
}},
ystaticy
reviewed
May 8, 2024
| return mark, false | ||
| } | ||
|
|
||
| func (m *MergeRanges) Covered(start []byte, end []byte) bool { |
There was a problem hiding this comment.
Suggested change
| func (m *MergeRanges) Covered(start []byte, end []byte) bool { | |
| // Covered returns true if m.ranges completely covers the startkey to endkey range. | |
| func (m *MergeRanges) Covered(start []byte, end []byte) bool { |
Signed-off-by: Ping Yu <[email protected]>
ystaticy
reviewed
May 8, 2024
txnkv/transaction/txn_file.go
Outdated
| ) | ||
|
|
||
| var ( | ||
| // BuildTxnFileMaxBackoff is max sleep time to build TxnFile. |
There was a problem hiding this comment.
Suggested change
| // BuildTxnFileMaxBackoff is max sleep time to build TxnFile. | |
| // BuildTxnFileMaxBackoff is max sleep Millisecond to build TxnFile. |
Signed-off-by: Ping Yu <[email protected]>
Signed-off-by: Ping Yu <[email protected]>
ystaticy
reviewed
May 8, 2024
| } | ||
|
|
||
| // chunkBatch.txnChunkSlice should be sorted by smallest. | ||
| func (b *chunkBatch) getSampleKeys() [][]byte { |
There was a problem hiding this comment.
How about rename to getSortedChunkSliceSmallestKeys, which better represents what the function returns.
There was a problem hiding this comment.
The method is named as its purpose, not implementation.
And the implementation would be changed in the future, e.g, we can store extra keys in the middle of chunk and use here, if necessary.
ystaticy
approved these changes
May 8, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Txn File transaction is used for transactions of big size (>= 16MiB). By building chunks on external worker and then ingesting to TiKV, txn file transactions will reduce cost and load of TiKV comparing to normal transactions.