Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Configurable Retention Period for Delta Snapshots #640

Conversation

seshachalam-yv
Copy link
Contributor

@seshachalam-yv seshachalam-yv commented Jun 28, 2023

What this PR does / why we need it:

This PR resolves the request for a configurable retention period for delta snapshots. This feature aims to provide users with more flexibility in preserving delta snapshots for both GC policies (Exponential and Limit-Based)

Changes introduced in this PR include:

  • Added a new delta-snapshot-retention-period parameter in the SnapshotterConfig to allow for the customization of the retention period for delta snapshots. This new setting can be adjusted via command-line flags.
  • Modified the GarbageCollectDeltaSnapshots function, formerly garbageCollectDeltaSnapshots, to utilize the DeltaSnapshotRetentionPeriod parameter when identifying delta snapshots for deletion. This change ensures that delta snapshots within the user-specified retention period will not be deleted.
  • Added documentation on the garbage collection mechanism (doc/usage/garbage_collection.md) to include a detailed explanation of the new delta-snapshot-retention-period setting.
  • Exported function GarbageCollectDeltaSnapshots to add unit tests.
  • Removed dead code from snapshotter_test.go

Which issue(s) this PR fixes:
Fixes #639

Special notes for your reviewer:

Release note:

Introduced `delta-snapshot-retention-period` CLI flag to extend the configurable retention period for delta snapshots in `etcd-backup-restore`, enhancing flexibility for backup retention.

This commit introduces a new parameter, `delta-snapshot-retention-period`, in the `SnapshotterConfig`, allowing users to customize the retention period for delta snapshots.

The garbage collection function `GarbageCollectDeltaSnapshots` is modified to use this new setting, ensuring delta snapshots within the user-specified period are not deleted.
@seshachalam-yv seshachalam-yv requested a review from a team as a code owner June 28, 2023 09:25
@gardener-robot gardener-robot added needs/review Needs review size/xl Size of pull request is huge (see gardener-robot robot/bots/size.py) needs/second-opinion Needs second review by someone else labels Jun 28, 2023
@gardener-robot-ci-2 gardener-robot-ci-2 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) needs/ok-to-test Needs approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Jun 28, 2023
@gardener-robot-ci-1 gardener-robot-ci-1 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jun 28, 2023
@gardener-robot-ci-2 gardener-robot-ci-2 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jun 28, 2023
@gardener-robot-ci-2 gardener-robot-ci-2 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jun 28, 2023
@gardener-robot-ci-3 gardener-robot-ci-3 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jun 28, 2023
@shreyas-s-rao shreyas-s-rao added this to the v0.26.0 milestone Jul 3, 2023
@gardener-robot-ci-1 gardener-robot-ci-1 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jul 4, 2023
@gardener-robot-ci-3 gardener-robot-ci-3 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jul 4, 2023
doc/usage/garbage_collection.md Outdated Show resolved Hide resolved
pkg/snapshot/snapshotter/garbagecollector.go Outdated Show resolved Hide resolved
@gardener-robot gardener-robot added the needs/changes Needs (more) changes label Jul 7, 2023
- Improved grammar and phrasing for better readability.
- Fixed doc string for 'GarbageCollectDeltaSnapshots' function.
@gardener-robot-ci-3 gardener-robot-ci-3 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Jul 12, 2023
Copy link
Contributor

@aaronfern aaronfern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing my comments @seshachalam-yv!
It looks good, just one last comment about the tests. Please address that

pkg/snapshot/snapshotter/snapshotter_test.go Outdated Show resolved Hide resolved
@gardener-robot-ci-3 gardener-robot-ci-3 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Jul 13, 2023
Co-authored-by: Aaron Francis Fernandes <[email protected]>

Update snapshotter_test.go with additional safety checks
@seshachalam-yv seshachalam-yv force-pushed the feature/configurable-delta-snapshot-retention branch from 4624166 to 89483ec Compare July 13, 2023 10:29
@gardener-robot-ci-3 gardener-robot-ci-3 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jul 13, 2023
@gardener-robot gardener-robot removed the needs/second-opinion Needs second review by someone else label Jul 13, 2023
@gardener-robot-ci-1 gardener-robot-ci-1 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Jul 13, 2023
Copy link
Collaborator

@shreyas-s-rao shreyas-s-rao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@seshachalam-yv thanks for the PR as well as detailed doc. I have few suggestions, but overall logic looks good.

pkg/snapshot/snapshotter/garbagecollector.go Show resolved Hide resolved
pkg/snapshot/snapshotter/snapshotter_test.go Show resolved Hide resolved
pkg/snapshot/snapshotter/snapshotter_test.go Outdated Show resolved Hide resolved
pkg/snapshot/snapshotter/snapshotter_test.go Outdated Show resolved Hide resolved
pkg/snapshot/snapshotter/snapshotter_test.go Outdated Show resolved Hide resolved
pkg/snapshot/snapshotter/snapshotter_test.go Outdated Show resolved Hide resolved
doc/usage/garbage_collection.md Show resolved Hide resolved
@gardener-robot gardener-robot added needs/second-opinion Needs second review by someone else and removed reviewed/lgtm Has approval for merging labels Jul 25, 2023
@gardener-robot-ci-1 gardener-robot-ci-1 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Jul 25, 2023
@gardener-robot-ci-3 gardener-robot-ci-3 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Jul 26, 2023
@seshachalam-yv seshachalam-yv force-pushed the feature/configurable-delta-snapshot-retention branch from 6d6eaf7 to 182d3dd Compare July 26, 2023 08:42
@gardener-robot-ci-3 gardener-robot-ci-3 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Jul 26, 2023
Copy link
Collaborator

@shreyas-s-rao shreyas-s-rao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@seshachalam-yv thank for making all the requested changes. No further comments.

Please create the two new issues as specified in the comments.

/lgtm

@seshachalam-yv
Copy link
Contributor Author

The integration test is failing due to a CI infrastructure issue. However, I have run it locally and it passed successfully, as shown in this test-output.txt. Consequently, I am proceeding to merge this PR.

@seshachalam-yv seshachalam-yv merged commit 37e073e into gardener:master Jul 27, 2023
1 check passed
@gardener-robot gardener-robot added the status/closed Issue is closed (either delivered or triaged) label Jul 27, 2023
@seshachalam-yv seshachalam-yv deleted the feature/configurable-delta-snapshot-retention branch July 27, 2023 16:40
seshachalam-yv added a commit to seshachalam-yv/etcd-backup-restore that referenced this pull request Oct 11, 2023
…#640)

* `feat: Implement configurable retention period for delta snapshots`

This commit introduces a new parameter, `delta-snapshot-retention-period`, in the `SnapshotterConfig`, allowing users to customize the retention period for delta snapshots.

The garbage collection function `GarbageCollectDeltaSnapshots` is modified to use this new setting, ensuring delta snapshots within the user-specified period are not deleted.

* Fixed lint error

* Fixed failing unit test

* Refactor and improve unit tests for GarbageCollectDeltaSnapshots

* Addressed PR feedback

- Improved grammar and phrasing for better readability.
- Fixed doc string for 'GarbageCollectDeltaSnapshots' function.

* Update snapshotter_test.go with additional safety checks

Co-authored-by: Aaron Francis Fernandes <[email protected]>

Update snapshotter_test.go with additional safety checks

* Apply suggestions from code review

Co-authored-by: Shreyas Rao <[email protected]>

* Addressed review feedback

---------

Co-authored-by: Shreyas Rao <[email protected]>
seshachalam-yv added a commit that referenced this pull request Oct 12, 2023
)

* `feat: Implement configurable retention period for delta snapshots`

This commit introduces a new parameter, `delta-snapshot-retention-period`, in the `SnapshotterConfig`, allowing users to customize the retention period for delta snapshots.

---------

Co-authored-by: Shreyas Rao <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs/ok-to-test Needs approval for testing (check PR in detail before setting this label because PR is run on CI/CD) needs/second-opinion Needs second review by someone else size/xl Size of pull request is huge (see gardener-robot robot/bots/size.py) status/closed Issue is closed (either delivered or triaged)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] Extend Retention Period for Delta Snapshots
7 participants