Skip to content
This repository has been archived by the owner on Jul 24, 2024. It is now read-only.

*: skip creating Domain for backup without stats #876

Merged
merged 10 commits into from
Mar 17, 2021

Conversation

overvenus
Copy link
Member

What problem does this PR solve?

This PR reduces memory usage during backup by skipping initializing Domain which loads all table info into memory.

Close #818

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
Manual tests
  1. Load 40k 40 fields YCSB tables
for i in $(seq 1 40000); do { go-ycsb load mysql -p workload=core -p mysql.host=172.16.5.33 -p mysql.port=47904 -p mysql.user=root -p recordcount=20 -p fieldcount=40 -p fieldlength=16 -p threadcount=4 -p table=usertable$i; } done;
  1. Backup 40k tables
br --pd http://PD_ADDR backup full -s "local:///path/to/backupdir"

The master branch BR takes about 3.09GB of memory.

This PR BR takes about 2.21G of memory.

Code changes

  • Has exported function/method change
  • Has exported variable/fields change

Related changes

  • Need to cherry-pick to the release branch

Release Note

  • Reduce memory usage during backup.

@overvenus overvenus changed the title *: skip creating Domain for backup withtout stats *: skip creating Domain for backup without stats Mar 15, 2021
@overvenus
Copy link
Member Author

/run-all-tests

1 similar comment
@overvenus
Copy link
Member Author

/run-all-tests

Copy link
Collaborator

@kennytm kennytm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest LGTM

pkg/backup/schema.go Outdated Show resolved Hide resolved
@YuJuncen
Copy link
Collaborator

/lgtm

@ti-srebot ti-srebot added the status/LGT1 LGTM1 label Mar 17, 2021
@kennytm
Copy link
Collaborator

kennytm commented Mar 17, 2021

/lgtm

@ti-srebot ti-srebot removed the status/LGT1 LGTM1 label Mar 17, 2021
@ti-srebot
Copy link
Contributor

/run-all-tests

@ti-srebot ti-srebot merged commit de96669 into pingcap:master Mar 17, 2021
ti-srebot pushed a commit to ti-srebot/br that referenced this pull request Mar 17, 2021
@ti-srebot
Copy link
Contributor

cherry pick to release-4.0 in PR #886

ti-srebot pushed a commit to ti-srebot/br that referenced this pull request Mar 17, 2021
@ti-srebot
Copy link
Contributor

cherry pick to release-5.0 in PR #887

overvenus added a commit that referenced this pull request Mar 18, 2021
overvenus added a commit that referenced this pull request Mar 18, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BuildBackupRangeAndSchema leaks memory
4 participants