Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stacks-inspect: add index-range option to replay-blocks #4885

Merged
merged 1 commit into from
Jun 17, 2024

Conversation

kantai
Copy link
Member

@kantai kantai commented Jun 14, 2024

Description

This adds the index-range option to replay-blocks. This is useful for selecting a uniformly random slice of blocks for replay (because ordering by index block hash is indistinguishable from a pseudorandom function).

Note: As part of this PR, I also considered making the replay-blocks command alter the SQLite transaction behavior from IMMEDIATE to DEFERRED. The goal was to allow multiple processes to share a chainstate directory during replay-blocks. Unfortunately, this didn't really help, because replay-blocks does perform writes in its transactions: it just rolls back those transactions. This means that SQLite will try to obtain a write lock even in DEFERRED operation.

However, one workable strategy for avoiding storage overhead when trying to run multiple process with replay-blocks is to copy the whole chainstate directory except for chainstate/vm/clarity/marf.sqlite.blobs, and then simply symlink that file for each instance of the replay-blocks processes. That file isn't a database, so it doesn't lock on transactions (and this is safe because none of the replay-blocks processes actually write to that file). This file is also ~75% of the chainstate's disk size.

@kantai kantai requested a review from a team as a code owner June 14, 2024 17:22
@jcnelson
Copy link
Member

@kantai what if you used BEGIN CONCURRENT instead of BEGIN DEFERRED?

From the docs:

Usually, SQLite allows at most one writer to proceed concurrently. The BEGIN CONCURRENT enhancement allows multiple writers to process write transactions simultanously if the database is in "wal" or "wal2" mode, although the system still serializes COMMIT commands.

When a write-transaction is opened with "BEGIN CONCURRENT", actually locking the database is deferred until a COMMIT is executed. This means that any number of transactions started with BEGIN CONCURRENT may proceed concurrently. The system uses optimistic page-level-locking to prevent conflicting concurrent transactions from being committed.

@kantai
Copy link
Member Author

kantai commented Jun 14, 2024

I thought about that @jcnelson, but I think BEGIN CONCURRENT is only available in a branch of sqlite (https://github.com/sqlite/sqlite/tree/begin-concurrent).

@saralab saralab requested a review from obycode June 17, 2024 15:10
Copy link
Contributor

@obycode obycode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@kantai kantai added this pull request to the merge queue Jun 17, 2024
Merged via the queue into develop with commit c6f055b Jun 17, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants