feat(exex): write ahead log #10995
Draft
feat(exex): write ahead log #10995
+846
−3
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
shekhirin
added
C-enhancement
shekhirin
force-pushed
the
alexey/exex-wal
branch
from
acf5263
to
213bcea
Compare
shekhirin
commented
Sep 20, 2024
shekhirin
force-pushed
the
alexey/exex-wal
branch
from
87f42f2
to
6c2874a
Compare
shekhirin
force-pushed
the
alexey/exex-wal
branch
from
6c2874a
to
b05bb87
Compare
shekhirin
force-pushed
the
alexey/exex-wal
branch
from
2db5ebd
to
d80c196
Compare
shekhirin
force-pushed
the
alexey/exex-wal
branch
from
99b0e65
to
fa17c65
Compare
shekhirin
force-pushed
the
alexey/exex-wal
branch
from
4a16fd6
to
1b79e58
Compare
shekhirin
force-pushed
the
alexey/exex-wal
branch
from
0cc1753
to
108981f
Compare
mattsse
reviewed
Sep 20, 2024
crates/exex/exex/src/wal/mod.rs
Outdated
| /// Clears the block cache and fills it with the notifications from the [`Storage`], up to the | ||
| /// given offset in bytes, not inclusive. | ||
| #[instrument(target = "exex::wal", skip(self))] | ||
| fn fill_block_cache(&mut self, to_offset: u64) -> eyre::Result<()> { |
There was a problem hiding this comment.
this is basically initialization?
Comment on lines
+10
to
+18
| /// The underlying WAL storage backed by a file. | ||
| /// | ||
| /// Each notification is written without any delimiters and structured as follows: | ||
| /// ```text | ||
| /// +--------------------------+----------------------------------+ | ||
| /// | little endian u32 length | MessagePack-encoded notification | | ||
| /// +--------------------------+----------------------------------+ | ||
| /// ``` | ||
| /// The length is the length of the MessagePack-encoded notification in bytes. |
There was a problem hiding this comment.
not obvious to me what the order is here,
are new notifications appended?
crates/exex/exex/src/wal/storage.rs
Outdated
Comment on lines
44
to
53
| /// Truncates the underlying file from the given byte offset (inclusive) to the end of the file. | ||
| /// | ||
| /// 1. Creates a new file and copies all notifications starting from the offset (inclusive) to | ||
| /// the end of the original file. | ||
| /// 2. Renames the new file to the original file. | ||
| /// | ||
| /// # Returns | ||
| /// | ||
| /// The old and new underlying file sizes in bytes | ||
| pub(super) fn truncate_from_offset(&mut self, offset: u64) -> eyre::Result<(u64, u64)> { |
There was a problem hiding this comment.
so finalizing a block means we must copy and reinitialize the entire file?
with > 64 blocks (more with reorged blocks) this isn't very cheap I suppose
what happens to the offsets that the cache tracks?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Towards #10489.
Implements the write-ahead log for ExEx notifications but doesn't integrate it yet.
reth/crates/exex/exex/src/wal/mod.rs
Lines 21 to 33 in 0cc1753