Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: migrate to sqlite for more consistent (and faster) storage #485

Closed
wants to merge 3 commits into from

Conversation

garethgeorge
Copy link
Owner

@garethgeorge garethgeorge commented Sep 20, 2024

Backrest's storage today is fairly special in that it's implemented on a key value store and augments that storage with it's own simple indexes. This is very fast, but comes with significant complexity and is not standard.

As Backrest has matured maintainability is increasingly a priority. It makes sense to look at migrating to a more standard embedded database e.g. SQLite which performs embarassingly well (with the caveat of higher constant overhead e.g. with very small get or list operations)...

In addition, SQLite's blob storage scales very well even to quite large object sizes. In this PR I've prototyped an approach to log storage that simply "sticks" the logs in the database. See https://www.sqlite.org/intern-v-extern-blob.html for a bit more context on the tradeoffs.

Insertion benchmarks

goos: linux
goarch: amd64
pkg: github.com/garethgeorge/backrest/internal/oplog/storetests
cpu: AMD Ryzen 9 5900X 12-Core Processor            
BenchmarkAdd/bbolt-24         	     746	   1430713 ns/op	   90699 B/op	     190 allocs/op
BenchmarkAdd/memory-24        	 2723769	      3822 ns/op	    1290 B/op	      30 allocs/op
BenchmarkAdd/sqlite-24        	    1304	    885453 ns/op	     758 B/op	      19 allocs/op
PASS
ok  	github.com/garethgeorge/backrest/internal/oplog/storetests	15.449s

Listing benchmarks

Running tool: /usr/local/go/bin/go test -benchmem -run=^$ -bench ^BenchmarkList$ github.com/garethgeorge/backrest/internal/oplog/storetests

goos: linux
goarch: amd64
pkg: github.com/garethgeorge/backrest/internal/oplog/storetests
cpu: AMD Ryzen 9 5900X 12-Core Processor            
BenchmarkList/100/bbolt-24   	   13281	     90255 ns/op	   40216 B/op	     922 allocs/op
BenchmarkList/100/bbolt_complexquery-24         	   12448	     95725 ns/op	   42504 B/op	     939 allocs/op
BenchmarkList/100/memory-24                     	   21810	     55301 ns/op	   28920 B/op	     303 allocs/op
BenchmarkList/100/memory_complexquery-24        	   21769	     56464 ns/op	   28920 B/op	     303 allocs/op
BenchmarkList/100/sqlite-24                     	    8440	    154960 ns/op	   36253 B/op	     720 allocs/op
BenchmarkList/100/sqlite_complexquery-24        	    6769	    174632 ns/op	   36571 B/op	     725 allocs/op
BenchmarkList/1000/memory-24                    	    1822	    598117 ns/op	  288219 B/op	    3003 allocs/op
BenchmarkList/1000/memory_complexquery-24       	    2221	    592119 ns/op	  288219 B/op	    3003 allocs/op
BenchmarkList/1000/sqlite-24                    	    1104	   1129790 ns/op	  353072 B/op	    7020 allocs/op
BenchmarkList/1000/sqlite_complexquery-24       	     806	   1509434 ns/op	  353393 B/op	    7025 allocs/op
BenchmarkList/1000/bbolt-24                     	    1212	    938114 ns/op	  408780 B/op	    9027 allocs/op
BenchmarkList/1000/bbolt_complexquery-24        	    1284	    968973 ns/op	  425519 B/op	    9045 allocs/op
BenchmarkList/10000/sqlite-24                   	     100	  11624762 ns/op	 3521098 B/op	   70020 allocs/op
BenchmarkList/10000/sqlite_complexquery-24      	      87	  14006409 ns/op	 3521406 B/op	   70025 allocs/op
BenchmarkList/10000/bbolt-24                    	     100	  10796012 ns/op	 5843476 B/op	  183450 allocs/op
BenchmarkList/10000/bbolt_complexquery-24       	     100	  11578388 ns/op	 6012059 B/op	  184427 allocs/op
BenchmarkList/10000/memory-24                   	     200	   5959604 ns/op	 2881982 B/op	   30003 allocs/op
BenchmarkList/10000/memory_complexquery-24      	     200	   5848090 ns/op	 2881982 B/op	   30003 allocs/op
PASS
ok  	github.com/garethgeorge/backrest/internal/oplog/storetests	54.054s

Note: this fix relocates the restic binary on windows to C:\Program Files\backrest  OR to the directory where backrest is installed (path relative).
@garethgeorge
Copy link
Owner Author

Note: actually seeing very poor performance out of the textstore implemented on sqlite. Some more thinking is needed to decide if this is the right way to go.

@garethgeorge
Copy link
Owner Author

Opting to contribute only the oplog store implementation for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant