Skip to content
This repository has been archived by the owner on Jul 24, 2024. It is now read-only.

pkg/utils: speed up LoadBackupMeta (#744) #745

Merged
merged 2 commits into from
Mar 1, 2021

Conversation

ti-srebot
Copy link
Contributor

@ti-srebot ti-srebot commented Feb 19, 2021

cherry-pick #744 to release-4.0
You can switch your code base to this Pull Request by using git-extras:

# In br repo:
git pr https://github.com/pingcap/br/pull/745

After apply modifications, you can push your change to this PR via:

git push [email protected]:ti-srebot/br.git pr/745:release-4.0-e901b9eddeb4

What problem does this PR solve?

Speed up LoadBackupMeta.

Cc #705

Manually tested with 39k tables, it improves fast checksum about 53 times.

Benchmark results
# master branch
[2021/02/19 14:50:58.551 +08:00] [INFO] [collector.go:60] ["Full backup Success summary: total backup ranges: 117738, total success: 117738, total failed: 0, total take(Full backup time): 2m43.027476154s, total take(real time): 23m31.342884193s, total kv: 117738000, total size(MB): 11298.47, avg speed(MB/s): 69.30"] ["backup checksum"=18.410789857s] ["backup fast checksum"=19m20.755603256s] ["backup total regions"=117824] [BackupTS=423025980821995522] [Size=4608796580]

# this branch
[2021/02/19 16:29:03.509 +08:00] [INFO] [collector.go:60] ["Full backup Success summary: total backup ranges: 117738, total success: 117738, total failed: 0, total take(Full backup time): 2m34.65939439s, total take(real time): 4m22.869289236s, total kv: 117738000, total size(MB): 11298.47, avg speed(MB/s): 73.05"] ["backup checksum"=18.999134215s] ["backup fast checksum"=22.223445159s] ["backup total regions"=117824] [Size=4608796580] [BackupTS=423027824506961921]

# master branch
PASS: schema_test.go:218: testSchemaSuite.BenchmarkLoadBackupMeta1024          1        2805279107 ns/op
PASS: schema_test.go:229: testSchemaSuite.BenchmarkLoadBackupMeta10240         1      428872156932 ns/op
PASS: schema_test.go:207: testSchemaSuite.BenchmarkLoadBackupMeta64          200           9807260 ns/op

# this branch
PASS: schema_test.go:218: testSchemaSuite.BenchmarkLoadBackupMeta1024         50          29454271 ns/op
PASS: schema_test.go:229: testSchemaSuite.BenchmarkLoadBackupMeta10240         1        1150945026 ns/op
PASS: schema_test.go:207: testSchemaSuite.BenchmarkLoadBackupMeta64         1000           1662240 ns/op

What is changed and how it works?

Check List

Tests

  • Unit test

Related changes

  • Need to cherry-pick to the release branch

Release Note

  • Improve backup performance when there are many tables.

@kennytm
Copy link
Collaborator

kennytm commented Feb 19, 2021

/lgtm

@ti-srebot ti-srebot added the status/LGT1 LGTM1 label Feb 19, 2021
@overvenus overvenus added this to the v4.0.12 milestone Feb 20, 2021
@ti-srebot ti-srebot added status/LGT2 LGTM2 and removed status/LGT1 LGTM1 labels Mar 1, 2021
@overvenus
Copy link
Member

/merge

@ti-srebot
Copy link
Contributor Author

/run-all-tests

@ti-srebot ti-srebot merged commit 22d421a into pingcap:release-4.0 Mar 1, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants