Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Restore zstd-compressed database backup #273

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

dlipovetsky
Copy link
Contributor

The database backup is compressed using zstd. This allows us to load the backup without having to decompress it beforehand.

Please note that this change is backward compatible: both compressed and uncompressed backups are supported. I have added an integration test (using an on-disk database) to ensure this.

@dlipovetsky
Copy link
Contributor Author

I removed an unused function, and force-pushed.

The database backup is compressed using zstd. This allows us to load the
backup without having to decompress it beforehand.
Comment on lines +29 to +40
err = db.Load(zr, runtime.NumCPU())
if errors.Is(err, zstd.ErrMagicMismatch) {
glog.V(2).Infof("database file is not compressed with zstd, will load without decompressing")

// We already loaded the database, advancing the file offset. To load the database again,
// we must reset the offset to the start.
if _, err := file.Seek(0, io.SeekStart); err != nil {
return errors.Wrapf(err, "failed to to rewind to start of database restore file: %q", filename)
}
err = db.Load(file, runtime.NumCPU())
}
if err != nil {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to check if the file is compressed or not before loading the database, but could not find a way to do that using the zstd package. Attempting to load the database is the best alternative I found.

@dlipovetsky
Copy link
Contributor Author

CI failure appears to be unrelated to this change.

=== RUN   Test_bigPictureWithExclusionRules
W0422 16:21:18.250823   19125 reflector.go:424] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: failed to list *unstructured.Unstructured: Get "http://url/apis/g/v1/things?limit=500&resourceVersion=0": dial tcp: lookup url: Temporary failure in name resolution
E0422 16:21:18.250924   19125 reflector.go:140] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: Get "http://url/apis/g/v1/things?limit=500&resourceVersion=0": dial tcp: lookup url: Temporary failure in name resolution
    kubewatcher_test.go:174: 
        	Error Trace:	/home/runner/work/sloop/sloop/pkg/sloop/ingress/kubewatcher_test.go:174
        	Error:      	Not equal: 
        	            	expected: 3
        	            	actual  : 2
        	Test:       	Test_bigPictureWithExclusionRules
--- FAIL: Test_bigPictureWithExclusionRules (1.00s)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant