chore: migrate to sqlite for more consistent (and faster) storage #485
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backrest's storage today is fairly special in that it's implemented on a key value store and augments that storage with it's own simple indexes. This is very fast, but comes with significant complexity and is not standard.
As Backrest has matured maintainability is increasingly a priority. It makes sense to look at migrating to a more standard embedded database e.g. SQLite which performs embarassingly well (with the caveat of higher constant overhead e.g. with very small get or list operations)...
In addition, SQLite's blob storage scales very well even to quite large object sizes. In this PR I've prototyped an approach to log storage that simply "sticks" the logs in the database. See https://www.sqlite.org/intern-v-extern-blob.html for a bit more context on the tradeoffs.
Insertion benchmarks
Listing benchmarks