add priv step1 #281
Merged
add priv step1 #281
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Prepare for grant statement.
| @@ -147,6 +147,10 @@ func TypeToStr(tp byte, binary bool) string { | |||
| return "timestamp" | |||
| case mysql.TypeBit: | |||
| return "bit" | |||
| case mysql.TypeEnum: | |||
There was a problem hiding this comment.
It's better. But I am blocked here and can not wait.
|
LGTM |
1 similar comment
|
LGTM |
YuJuncen
pushed a commit
to YuJuncen/tidb
that referenced
this pull request
Apr 23, 2021
* restore-util: Implement split/scatter (pingcap#274) * implement split/scatter Signed-off-by: 5kbpers <[email protected]> * init test Signed-off-by: 5kbpers <[email protected]> * redesign output/input of the lib Signed-off-by: 5kbpers <[email protected]> * update dependency Signed-off-by: 5kbpers <[email protected]> * add commments and more tests Signed-off-by: 5kbpers <[email protected]> * add ScanRegions interface to Client Signed-off-by: 5kbpers <[email protected]> * fix potential data race Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Co-Authored-By: kennytm <[email protected]> * Update pkg/restore-util/client.go Co-Authored-By: kennytm <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * update dependency Signed-off-by: 5kbpers <[email protected]> * resolve conflicts Signed-off-by: 5kbpers <[email protected]> * fix prefix rewrite Signed-off-by: 5kbpers <[email protected]> * add RewriteRule/skip failed scatter region/retry the SplitRegion Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> * check if region has peer Signed-off-by: 5kbpers <[email protected]> * more logs Signed-off-by: 5kbpers <[email protected]> * restore-util: add split retry interval (pingcap#277) * reset dependencies to release-3.1 * add split retry interval Signed-off-by: 5kbpers <[email protected]> * fix go.sum Signed-off-by: 5kbpers <[email protected]> * restore-util: wait for scatter region sequentially (pingcap#279) * wait for scatter region sequentially Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * restore-util: add on split hook (pingcap#281) * restore-util: add on split hook Signed-off-by: Neil Shen <[email protected]> * Nil check onSplit Co-Authored-By: kennytm <[email protected]> * restore-util: fix returned new region is nil (pingcap#283) * restore-util: fix returned new region is nil Signed-off-by: 5kbpers <[email protected]> * more logs Signed-off-by: 5kbpers <[email protected]> * *: gofmt Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Co-Authored-By: kennytm <[email protected]> * fix log Signed-off-by: 5kbpers <[email protected]> * restore-util: call onSplit on splitByRewriteRules (pingcap#285) Signed-off-by: Neil Shen <[email protected]> * restore-util: fix overlapped error message (pingcap#293) * restore-util: fix overlapped error message Signed-off-by: 5kbpers <[email protected]> * fix log message Signed-off-by: 5kbpers <[email protected]> * reduce error trace Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * restore-util: log warning when cannot find matched rewrite rule (pingcap#299) * restore-util: add method to set placement rules and store labels (pingcap#301) * restore-util: add method to set placement rules and store labels Signed-off-by: disksing <[email protected]> * minor fix Signed-off-by: disksing <[email protected]> * address comment Signed-off-by: disksing <[email protected]> * add GetPlacementRules Signed-off-by: disksing <[email protected]> * fix test Signed-off-by: disksing <[email protected]> * restore-util: support batch split (pingcap#300) * restore-util: support batch split Signed-off-by: 5kbpers <[email protected]> * go fmt Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Co-Authored-By: kennytm <[email protected]> * address commits Signed-off-by: 5kbpers <[email protected]> * Update pkg/restore-util/split.go Co-Authored-By: kennytm <[email protected]> * add onSplit callback Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * restore-util: add upper bound time for waiting for scatter (pingcap#305) * restore: fix scatter regions failed Signed-off-by: 5kbpers <[email protected]> * add log Signed-off-by: 5kbpers <[email protected]> * stop waiting for scatter after 3min Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * restore-util: fix wrong url (pingcap#306) Signed-off-by: disksing <[email protected]> * restore-util: add warning about unmatched table id (pingcap#313) * restore-util: support table partition Signed-off-by: 5kbpers <[email protected]> * fix log Signed-off-by: 5kbpers <[email protected]> * warn table id does not match Signed-off-by: 5kbpers <[email protected]> * add unit tests Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Co-Authored-By: Neil Shen <[email protected]> * fix compile error Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> Co-authored-by: Ian <[email protected]> Co-authored-by: Neil Shen <[email protected]> * *: prune tidb-tools Signed-off-by: Neil Shen <[email protected]> * restore: address linters suggestions Signed-off-by: Neil Shen <[email protected]> * restore: merge restoreutil into restore Signed-off-by: Neil Shen <[email protected]> * address comment Signed-off-by: Neil Shen <[email protected]> Co-authored-by: 5kbpers <[email protected]> Co-authored-by: kennytm <[email protected]> Co-authored-by: disksing <[email protected]> Co-authored-by: Ian <[email protected]>
YuJuncen
pushed a commit
to YuJuncen/tidb
that referenced
this pull request
Apr 23, 2021
* lightning: split large csv file if possible
* gofmt
* gofmt
* unit test
* add unit test
* tiny change
* tiny refine
* fix ci
* remove useless code
* fix ci
* fix ci
* address comments
* go fmt for all
* Replace CSV Ragel parser by a hand-written parser copied from encoding/csv
Conflicts:
lightning/mydump/csv_parser_generated.go
* fix conflict
* update
* update again
* send a batch of kv in encodeLoop
* use sync.Pool
* Close channel instead of push one entry.
* Use copy instead append
* Fix test and failpoint version
* Reuse slice of record
This expected to avoid about 3.5% of alloc_objects
alloc_objects:
Total: 773496750 773873722 (flat, cum) 7.18%
177 . . parser.fieldIndexes = parser.fieldIndexes[:0]
178 . .
179 . . isEmptyLine := true
...
225 386621314 386621314 str := string(parser.recordBuffer) // Convert to string once to batch allocations
226 386875436 386875436 dst := make([]string, len(parser.fieldIndexes))
* Use pool for mutation
This take most alloc in WriteRows:
ROUTINE ======================== github.com/pingcap/tidb-lightning/lightning/backend.(*importer).WriteRows in /Users/huangjiahao/go/src/github.com/pingcap/tidb-lightning/lightning/backend/importer.go
797370418 980241246 (flat, cum) 9.09% of Total
. . 155: kvs := rows.(kvPairs)
...
...
. . 192: for i, pair := range kvs {
772641868 772641868 193: mutations[i] = &kv.Mutation{
. . 194: Op: kv.Mutation_Put,
. . 195: Key: pair.Key,
. . 196: Value: pair.Val,
. . 197: }
. . 198: }
* Set GC percent as 500 default
Lightning allocates too many transient objects and heap size is small,
so garbage collections happen too frequently and lots of time is spent in GC component.
In a test of loading the table `order_line.csv` of 14k TPCC.
The time need of `encode kv data and write` step reduce from 52m4s to 37m30s when change
GOGC from 100 to 500, the total time needed reduce near 15m too.
The cost of this is the memory of lightnin at runtime grow from about 200M to 700M, but it's acceptable.
So we set the gc percentage as 500 default to reduce the GC frequency instead of 100.
* Remove MaxKVPairs in Mydump
has been move to Importer part
* Remove outdate code
* Update tidb version
For pingcap@495f8b7
disable UpdateDeltaForTable if TxnCtx is nil
* Address comment
* Remain append
Co-authored-by: xuhuaiyu <[email protected]>
xhebox
pushed a commit
to xhebox/tidb
that referenced
this pull request
Sep 28, 2021
xhebox
pushed a commit
to xhebox/tidb
that referenced
this pull request
Oct 8, 2021
ti-chi-bot
pushed a commit
that referenced
this pull request
Oct 9, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Prepare for grant statement.