Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add method to RocksDB for backing up partitions #304

Merged
merged 30 commits into from
Jul 19, 2022

Conversation

wbarnha
Copy link
Member

@wbarnha wbarnha commented May 14, 2022

I have a specific use-case where I need to actively backup a RocksDB store. I've had success using rocksdb.BackupEngine with faust but it'd be much more usable if the code was integrated into faust.

I should note I've only had success with this using librocksdb-dev==6.11.4-3 on Ubuntu 22.04 and the python-rocksdb bindings from https://github.com/NightTsarina/python-rocksdb.

If you have interest in this, let me know because I really need this feature.

@wbarnha wbarnha changed the title Add method to RocksDB for backing up partitions Draft: Add method to RocksDB for backing up partitions May 14, 2022
@wbarnha wbarnha changed the title Draft: Add method to RocksDB for backing up partitions WIP: Add method to RocksDB for backing up partitions May 14, 2022
@wbarnha wbarnha changed the title WIP: Add method to RocksDB for backing up partitions Add method to RocksDB for backing up partitions May 14, 2022
@wbarnha
Copy link
Member Author

wbarnha commented May 18, 2022

I'm satisfied with the changes I have now. If you have any other changes you'd like to see, let me know. I haven't done any testing for Aerospike, but it seems possible to backup a table from what's done in https://github.com/aerospike/aerospike-tools-backup/blob/fc348e6016cdfccc0e75a6b76621067e57d2a11c/test/integration/run_backup.py.

@wbarnha
Copy link
Member Author

wbarnha commented May 18, 2022

I forgot to test some last-minute changes earlier. I've tested everything and runs properly now.

@wbarnha
Copy link
Member Author

wbarnha commented May 19, 2022

Removed import of BackupEngine to pass lint check, should be good now...

@wbarnha
Copy link
Member Author

wbarnha commented May 27, 2022

I'm not sure where this should be added, but according to https://fauststream.slack.com/archives/C7FL9CB5E/p1594816114113200?thread_ts=1594815786.109000&cid=C7FL9CB5E, recoveries are automatically triggered when a table is accessed. After testing further, it seems that calling restore_backup is equally safe as calling it during initialization or in asynchronous tasks that startup upon app initialization. Still haven't tested periodically restoring backups while the app is initialized since I haven't needed to do that.

@wbarnha
Copy link
Member Author

wbarnha commented Jun 2, 2022

On further testing, it appears that the current release of https://github.com/faust-streaming/python-rocksdb is fine for accessing the backup functions in librocksdb-dev. For older versions that are released such as 5.8.8, I got errors along the lines of rocksdb.error.NotImplementedError. We may want to add a disclaimer that the backup options are only available in specific versions of librocksdb.

@codecov-commenter
Copy link

codecov-commenter commented Jul 19, 2022

Codecov Report

Merging #304 (836751a) into master (a887571) will decrease coverage by 0.24%.
The diff coverage is 40.81%.

@@            Coverage Diff             @@
##           master     #304      +/-   ##
==========================================
- Coverage   94.66%   94.42%   -0.25%     
==========================================
  Files         100      100              
  Lines       10855    10903      +48     
  Branches     1520     1529       +9     
==========================================
+ Hits        10276    10295      +19     
- Misses        505      533      +28     
- Partials       74       75       +1     
Impacted Files Coverage Δ
faust/stores/rocksdb.py 89.38% <36.11%> (-6.98%) ⬇️
faust/stores/aerospike.py 76.12% <50.00%> (-0.70%) ⬇️
faust/tables/objects.py 97.70% <50.00%> (-2.30%) ⬇️
faust/stores/memory.py 95.23% <60.00%> (-4.77%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a887571...836751a. Read the comment docs.

@wbarnha
Copy link
Member Author

wbarnha commented Jul 19, 2022

I'd like to squash a lot of these commits before merging... unless the maintainers have the option to automatically do that 😄

@patkivikram patkivikram merged commit 0bb2685 into faust-streaming:master Jul 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants