Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: support for begin concurrent #1631

Open
aboodman opened this issue Aug 3, 2024 · 0 comments
Open

Feature request: support for begin concurrent #1631

aboodman opened this issue Aug 3, 2024 · 0 comments

Comments

@aboodman
Copy link

aboodman commented Aug 3, 2024

We are using the begin concurrent branch (https://www.sqlite.org/src/vdiff?branch=begin-concurrent -- see also https://www.sqlite.org/cgi/src/doc/begin-concurrent/doc/begin_concurrent.md) of sqlite for https://zerosync.dev.

We use it in kind of an interesting way: we don't ever commit the concurrent transactions. Instead, we open concurrent write transactions, read and write data to them, then rollback the transaction.

The reason we want to do this is because to accomplish its work, zero needs to maintain materialized views of queries from the client. While cheaper than running each client query from scratch, maintaining these views can still be expensive so we want to do it on multiple threads.

The algorithm we have developed to maintain these views requires a series of interactive write/read steps. For an incoming write transaction, we write a row to the datastore (sqlite), do some queries, write another row, do some queries, etc. At the end of these series of steps we have the output that we send to the client. We cannot do all the writes up front - the correctness of the algorithm requires the write/read interleaving.

If we run this algorithm on multiple thread in parallel, the writes will take eager locks and conflict with each other. But begin concurrent give us an awesome out: we can open transactions in parallel, do our write/read pairs, then abort the transactions. Then we just have a single 'writer' transaction that actually flushes the persistent writes to the db at the end.

Unfortunately we have to use this kind of semi-zombie branch of sqlite to accomplish this. It would be super useful to have it officially supported in libsql.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant