Restore: implement restorer for compacted SST/Snapshot/log files #57208

3pointer · 2024-11-07T10:12:42Z

What problem does this PR solve?

Issue Number: close #57209

Problem Summary:

This pull request introduces a unified structure for managing compacted SST, Snapshot, and log files. Previously, these file types were handled independently, leading to redundant code and potential inconsistencies. The new common struct simplifies file management, improves code maintainability, and reduces the likelihood of errors.

What changed and how does it work?

Unified Struct Abstraction: A shared struct has been created to manage compacted SST, Snapshot, and log files collectively, providing a consistent interface across these file types.

// FileRestorer defines the essential methods required for restoring SST files, 
// covering different types of backups and compacted logs, including:
// 1. Raw backup SSTs
// 2. Txn backup SSTs
// 3. TiDB backup SSTs
// 4. Compacted log SSTs
// 
// Base structures for these restorations include:
// - SimpleRestorer: Handles raw, txn, and compacted log backups.
// - MultiTablesRestorer: Manages snapshot-based multi-table restorations.
type FileRestorer interface {
	// Restore imports files to TiKV.
	Restore(onProgress func(int64), files ...BatchRestoreFilesInfo) error
	// WaitUntilFinish waits for all pending restore files to complete.
	WaitUntilFinish() error
	// Close releases resources.
	Close() error
}

Impact: This update significantly streamlines the restoration process, reducing complexity and improving performance by adopting a unified handling approach. Components across the system have been updated to leverage the new struct, fostering consistency and reducing maintenance overhead.
NOTE: This PR doesn't consider the checkpoint logic for compacted log files. I'll do it in another PR.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

ti-chi-bot · 2024-11-07T10:12:47Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

ti-chi-bot · 2024-11-07T10:12:54Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from 3pointer, ensuring that each of them provides their approval before proceeding. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

br/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tiprow · 2024-11-07T10:13:16Z

Hi @3pointer. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2024-11-12T10:19:21Z

Codecov Report

Attention: Patch coverage is 48.05195% with 560 lines in your changes missing coverage. Please review.

Project coverage is 74.6477%. Comparing base (4a6bf46) to head (1f1b636).

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #57208        +/-   ##
================================================
+ Coverage   72.8367%   74.6477%   +1.8110%     
================================================
  Files          1672       1721        +49     
  Lines        462640     471130      +8490     
================================================
+ Hits         336972     351688     +14716     
+ Misses       104878      97343      -7535     
- Partials      20790      22099      +1309

Flag	Coverage Δ
integration	`49.0657% <48.0519%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.7673% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`59.8005% <48.0519%> (+14.6963%)`	⬆️

YuJuncen

rest lgtm

YuJuncen · 2024-11-14T07:50:20Z

br/pkg/restore/log_client/compacted_file_strategy.go

+	}
+	if len(sstOutputs) != len(subCompaction.SstOutputs) {
+		log.Info("partial files in sub compaction skipped due to checkpoint")
+		subCompaction.SstOutputs = sstOutputs


It is a little strange that a predicate function modifies its argument. Would you add some comments in the interface?

This is because the subCompaction is not a basic restore unit during restore, the restore unit is file. and subCompaction might contains multiple files. sometimes we may need to skip partial files in one subCompaction.

br/pkg/restore/log_client/log_file_manager_test.go

YuJuncen · 2024-11-14T08:10:01Z

br/pkg/restore/split/splitter.go

+	// Skip the file by checkpoints or invalid files
+	ShouldSkip(T) bool
+	// GetAccumulations returns an iterator for the accumulated values.
+	GetAccumulations() *SplitHelperIterator


It is a little strange that Accumulate receives T but GetAccumulations returns Valued? It seems this method actually returns the keys to be splitted.

Yes, I want to refactor SplitHelperIterator too, but I suggest doing this in future PR.

br/pkg/restore/split/splitter.go

YuJuncen · 2024-11-14T08:59:16Z

br/pkg/restore/restorer.go

+	Close() error
+}
+
+type FileImporter interface {


It seems FileImporter is almost the same as SstRestorer. Would you merge them or add some comments that explains why we need two?

ok, add some comments, PTAL

br/pkg/restore/restorer.go

YuJuncen · 2024-11-14T09:09:10Z

br/pkg/restore/restorer.go

+					onProgress(int64(len(filesReplica)))
+				}
+			}()
+			if importErr := m.fileImporter.Import(m.ectx, filesReplica...); importErr != nil {


It seems we already managed the concurrenct execution. Why we still need ConcurrentlFileImporter here? Would you replace it with a normal one or add some comment here to explain?

ConcurrentFileImporter only used in MultiTablesRestorer, which has a condition wait during restore. this make a better control of concurrency to make download balanced.

If we make this a default behavior for raw/txn ssts. Then it can be removed. but in this PR, I want to make the behavior keeps same as before.

I'm wondering that given there is internal concurrency in the ConcurrentFileImporter, is the outer workerPool still needed?

YuJuncen · 2024-11-14T09:14:29Z

br/pkg/restore/restorer.go

+	return f[:idx]
+}
+
+type PipelineRestorerWrapper[T any] struct {


Perhaps this can be a plain function. So the type argument can be deduced, no explicit annotations like restore.PipelineRestorerWrapper[*logclient.LogDataFileInfo] needed.

br/pkg/restore/split/splitter.go

3pointer · 2024-11-17T14:06:43Z

/test unit-test

ti-chi-bot · 2024-11-17T14:06:46Z

@3pointer: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test build
/test check-dev
/test check-dev2
/test mysql-test
/test pull-br-integration-test
/test pull-integration-ddl-test
/test pull-lightning-integration-test
/test pull-mysql-client-test
/test pull-unit-test-ddlv1
/test unit-test

The following commands are available to trigger optional jobs:

/test pingcap/tidb/canary_ghpr_unit_test
/test pull-common-test
/test pull-e2e-test
/test pull-integration-common-test
/test pull-integration-copr-test
/test pull-integration-e2e-test
/test pull-integration-jdbc-test
/test pull-integration-mysql-test
/test pull-integration-nodejs-test
/test pull-integration-python-orm-test
/test pull-sqllogic-test
/test pull-tiflash-test

Use /test all to run the following jobs that were automatically triggered:

pingcap/tidb/ghpr_build
pingcap/tidb/ghpr_check
pingcap/tidb/ghpr_check2
pingcap/tidb/ghpr_mysql_test
pingcap/tidb/ghpr_unit_test
pingcap/tidb/pull_br_integration_test
pingcap/tidb/pull_integration_ddl_test
pingcap/tidb/pull_lightning_integration_test
pingcap/tidb/pull_mysql_client_test

In response to this:

/test unit-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

3pointer · 2024-11-17T14:06:48Z

/test pull-lightning-integration-test

tiprow · 2024-11-17T14:07:06Z

@3pointer: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test unit-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tiprow · 2024-11-17T14:07:10Z

@3pointer: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test pull-lightning-integration-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ti-chi-bot · 2024-11-17T15:14:47Z

@3pointer: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-br-integration-test	`fafb0d2`	link	true	`/test pull-br-integration-test`
idc-jenkins-ci-tidb/unit-test	`1f1b636`	link	true	`/test unit-test`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

restore: implement restorer for compacted SST/Snapshot/log files

1783421

ti-chi-bot bot added do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Nov 7, 2024

ti-chi-bot bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Nov 7, 2024

ti-chi-bot bot added do-not-merge/needs-triage-completed and removed do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked labels Nov 7, 2024

3pointer added 6 commits November 7, 2024 18:25

resolve conflicts

d449c73

Merge branch 'master' into feat/compacted_logs_2nd

5fab677

refine code after merge master

73959f4

add test cases

c6d643a

rename some names && more test cases

3674e0e

remove useless code

7ef0f7d

3pointer marked this pull request as ready for review November 12, 2024 09:59

ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 12, 2024

3pointer added 4 commits November 13, 2024 11:11

fix build

23a5d18

polish code

0ec3cb2

add NewSnapFileImporterOptions

c775e12

add tryMap for br iter

329bf69

ti-chi-bot bot removed the do-not-merge/needs-triage-completed label Nov 14, 2024

3pointer added 2 commits November 14, 2024 11:54

fix bazel

4dcd764

fix integration tests

ed5d95e

YuJuncen reviewed Nov 14, 2024

View reviewed changes

address comments

57d15db

3pointer added 10 commits November 15, 2024 18:00

address comments

57191e0

fix test

fbb71a9

fix integration test

3c0f79e

fix br_txn

886b8f9

fix br_raw

0784b43

refine code

a565ed0

rename some functions

b715725

fix reset limit

f05b60e

update case

5e2eac4

update cases

fafb0d2

3pointer added 2 commits November 17, 2024 22:19

Merge branch 'master' into feat/compacted_logs_2nd

d182593

fix the test after merge master

1f1b636

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restore: implement restorer for compacted SST/Snapshot/log files #57208

Restore: implement restorer for compacted SST/Snapshot/log files #57208

3pointer commented Nov 7, 2024 •

edited

Loading

ti-chi-bot bot commented Nov 7, 2024

ti-chi-bot bot commented Nov 7, 2024

tiprow bot commented Nov 7, 2024

codecov bot commented Nov 12, 2024 •

edited

Loading

YuJuncen left a comment

YuJuncen Nov 14, 2024

3pointer Nov 14, 2024

YuJuncen Nov 14, 2024

3pointer Nov 14, 2024

YuJuncen Nov 14, 2024

3pointer Nov 14, 2024

YuJuncen Nov 14, 2024

3pointer Nov 14, 2024

YuJuncen Nov 15, 2024

YuJuncen Nov 14, 2024

3pointer commented Nov 17, 2024

ti-chi-bot bot commented Nov 17, 2024

3pointer commented Nov 17, 2024

tiprow bot commented Nov 17, 2024

tiprow bot commented Nov 17, 2024

ti-chi-bot bot commented Nov 17, 2024

Restore: implement restorer for compacted SST/Snapshot/log files #57208

Are you sure you want to change the base?

Restore: implement restorer for compacted SST/Snapshot/log files #57208

Conversation

3pointer commented Nov 7, 2024 • edited Loading

What problem does this PR solve?

Problem Summary:

What changed and how does it work?

Check List

Release note

ti-chi-bot bot commented Nov 7, 2024

ti-chi-bot bot commented Nov 7, 2024

tiprow bot commented Nov 7, 2024

codecov bot commented Nov 12, 2024 • edited Loading

Codecov Report

YuJuncen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

3pointer commented Nov 17, 2024

ti-chi-bot bot commented Nov 17, 2024

3pointer commented Nov 17, 2024

tiprow bot commented Nov 17, 2024

tiprow bot commented Nov 17, 2024

ti-chi-bot bot commented Nov 17, 2024

3pointer commented Nov 7, 2024 •

edited

Loading

codecov bot commented Nov 12, 2024 •

edited

Loading