Incremental backup: fix calculation of binlog files to use #13066

shlomi-noach · 2023-05-10T14:14:44Z

Description

Incremental backup logic introduced in #11097 (still undocumented) did not evaluate the correct list of binary logs when gtid_purged happened to be exactly the last full backup position. There are probably more scenarios affected by the bug.

This PR treats gtid_purged and the last known backup pos correctly. Tests added.

Incremental backup and Point in time recovery as introduced in #11097, are still undocumented.

Related Issue(s)

Checklist

"Backport to:" labels have been added if this change should be back-ported
Tests were added or are not required
Did the new or modified tests pass consistently locally and on the CI
Documentation was added or is not required

Deployment Notes

Signed-off-by: Shlomi Noach <[email protected]>

vitess-bot · 2023-05-10T14:14:48Z

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

Ensure that the Pull Request has a descriptive title.
If this is a change that users need to know about, please apply the release notes (needs details) label so that merging is blocked unless the summary release notes document is included.
If a test is added or modified, there should be a documentation on top of the test to explain what the expected behavior is what the test does.

If a new flag is being introduced:

Is it really necessary to add this flag?
Flag names should be clear and intuitive (as far as possible)
Help text should be descriptive.
Flag names should use dashes (-) as word separators rather than underscores (_).

If a workflow is added or modified:

Each item in Jobs should be named in order to mark it as required.
If the workflow should be required, the maintainer team should be notified.

Bug fixes

There should be at least one unit or end-to-end test.
The Pull Request description should include a link to an issue that describes the bug.

Non-trivial changes

There should be some code comments as to why things are implemented the way they are.

New/Existing features

Should be documented, either by modifying the existing documentation or creating new documentation.
New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

Protobuf changes should be wire-compatible.
Changes to _vt tables and RPCs need to be backward compatible.
vtctl command output order should be stable and awk-able.
RPC changes should be compatible with vitess-operator
If a flag is removed, then it should also be removed from VTop, if used there.

mattlord

LGTM! Only had a few very minor nits/suggestions. I'll leave those up to you.

mattlord · 2023-05-10T19:51:09Z

go/vt/mysqlctl/binlogs_gtid.go

-		if unionPreviousGTIDs {
-			prevPos.GTIDSet = prevGTIDsUnion
+		if i == 0 {
+			return nil, "", "", vterrors.Errorf(vtrpc.Code_FAILED_PRECONDITION, "the very first binlog file %v has PreviousGTIDs %s that exceed given incremental backup pos. There are GTID entries that are missing and this backup cannot run", binlog, previousGTIDsPos)


Might be worth using the GTID_PURGED language/variable specifically to aid in debugging and bridging the MySQL<->Vitess concepts/pieces here.

go/vt/mysqlctl/binlogs_gtid.go

mattlord · 2023-05-10T19:54:37Z

go/vt/mysqlctl/binlogs_gtid.go

-		if i == 0 {
-			return nil, "", "", vterrors.Errorf(vtrpc.Code_FAILED_PRECONDITION, "the very first binlog file %v has PreviousGTIDs %s that exceed given incremental backup pos. There are GTID entries that are missing and this backup cannot run", binlog, prevPos)
+		if backupFromGTIDSet.Contains(previousGTIDsPos.GTIDSet) {
+			// All previous binary logs are fully contained by backupPos. So definitely all binlogs _prior_ to


IMO it's easier to reason about when explained as the current binary log being the oldest one available -- so all previous GTIDs have been purged and are no longer available.

mattlord · 2023-05-10T19:55:22Z

go/vt/mysqlctl/binlogs_gtid_test.go

+			},
+			backupPos:   "16b1039f-22b6-11ed-b765-0a43f95f28a3:1-63",
+			gtidPurged:  "16b1039f-22b6-11ed-b765-0a43f95f28a3:1-2",
+			expectError: "There are GTID entries that are missing",


Nit, but [purged and] no longer available feels more accurate than missing.

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach · 2023-05-11T05:54:38Z

@mattlord please see if rewording makes more sense. I find it really difficult to explain the constraints in proper English. Please feel free to edit.

mattlord · 2023-05-11T12:17:56Z

@mattlord please see if rewording makes more sense. I find it really difficult to explain the constraints in proper English. Please feel free to edit.

@shlomi-noach LGTM! ❤️

deepthi

Only nits on wording etc. The logic looks good to me.

go/vt/mysqlctl/binlogs_gtid.go

deepthi · 2023-05-17T16:19:49Z

go/vt/mysqlctl/binlogs_gtid.go

-			return nil, "", "", vterrors.Errorf(vtrpc.Code_FAILED_PRECONDITION, "binary log %v with previous GTIDS %s neither contains requested GTID %s nor contains it. Backup cannot take place", binlog, prevPos.GTIDSet, lookFromGTIDSet)
+		if !prevGTIDsUnion.Union(purgedGTIDSet).Contains(backupFromGTIDSet) {
+			return nil, "", "", vterrors.Errorf(vtrpc.Code_FAILED_PRECONDITION,
+				"Mismatching GTID entries. Requested backup pos has entries not found in the binary logs, and binary logs have entries not found in the requested backup pos. Requested pos=%v, binlog pos=%v",


Is it and binary logs or or binary logs?
In other words, does the check imply both conditions have occurred, or that at least one of them has occurred?
You could replace and with and/or.

It's an and, as described. It's like a Venn diagram of two circles which overlap, but neither fully contains the other.

Added Neither fully contains the other to the message. I hope it clarifies.

Co-authored-by: Deepthi Sigireddi <[email protected]> Signed-off-by: Shlomi Noach <[email protected]>

Incremental backup: fix calculation of binlog files to use

ba4de18

Signed-off-by: Shlomi Noach <[email protected]>

shlomi-noach added Type: Bug Component: Backup and Restore labels May 10, 2023

shlomi-noach requested review from deepthi, mattlord and rsajwani as code owners May 10, 2023 14:14

vitess-bot bot added NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsWebsiteDocsUpdate What it says labels May 10, 2023

shlomi-noach mentioned this pull request May 10, 2023

Bug Report: Incremental backup not working as described #13060

Closed

github-actions bot added this to the v17.0.0 milestone May 10, 2023

shlomi-noach removed NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsWebsiteDocsUpdate What it says labels May 10, 2023

mattlord approved these changes May 10, 2023

View reviewed changes

rewording

a05bc1d

Signed-off-by: Shlomi Noach <[email protected]>

deepthi approved these changes May 17, 2023

View reviewed changes

shlomi-noach and others added 3 commits May 18, 2023 06:22

Update go/vt/mysqlctl/binlogs_gtid.go

5b85909

Co-authored-by: Deepthi Sigireddi <[email protected]> Signed-off-by: Shlomi Noach <[email protected]>

clarify error message

c659eaf

Merge branch 'main' into backup-pitr-bug-first-binlog

e7c0515

shlomi-noach merged commit 2a69ceb into vitessio:main May 21, 2023

shlomi-noach deleted the backup-pitr-bug-first-binlog branch May 21, 2023 05:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incremental backup: fix calculation of binlog files to use #13066

Incremental backup: fix calculation of binlog files to use #13066

shlomi-noach commented May 10, 2023 •

edited by mattlord

Loading

vitess-bot bot commented May 10, 2023

mattlord left a comment

mattlord May 10, 2023

shlomi-noach May 11, 2023

mattlord May 10, 2023

mattlord May 10, 2023

shlomi-noach commented May 11, 2023

mattlord commented May 11, 2023

deepthi left a comment

deepthi May 17, 2023

shlomi-noach May 18, 2023

shlomi-noach May 18, 2023

Incremental backup: fix calculation of binlog files to use #13066

Incremental backup: fix calculation of binlog files to use #13066

Conversation

shlomi-noach commented May 10, 2023 • edited by mattlord Loading

Description

Related Issue(s)

Checklist

Deployment Notes

vitess-bot bot commented May 10, 2023

Review Checklist

General

If a new flag is being introduced:

If a workflow is added or modified:

Bug fixes

Non-trivial changes

New/Existing features

Backward compatibility

mattlord left a comment

Choose a reason for hiding this comment

mattlord May 10, 2023

Choose a reason for hiding this comment

shlomi-noach May 11, 2023

Choose a reason for hiding this comment

mattlord May 10, 2023

Choose a reason for hiding this comment

mattlord May 10, 2023

Choose a reason for hiding this comment

shlomi-noach commented May 11, 2023

mattlord commented May 11, 2023

deepthi left a comment

Choose a reason for hiding this comment

deepthi May 17, 2023

Choose a reason for hiding this comment

shlomi-noach May 18, 2023

Choose a reason for hiding this comment

shlomi-noach May 18, 2023

Choose a reason for hiding this comment

shlomi-noach commented May 10, 2023 •

edited by mattlord

Loading