[Bifrost] Improve merge operator performance by coalescing metadata updates #1726
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Jul 18, 2024
AhmedSoliman
force-pushed
the
pr1726
branch
2 times, most recently
from
eb65adf
to
1f010ea
Compare
AhmedSoliman
force-pushed
the
pr1726
branch
5 times, most recently
from
9f32462
to
399933d
Compare
tillrohrmann
approved these changes
Jul 22, 2024
A restructure the allows finer-grain control over what types loglet providers have access to. This will be exploited more in upcoming PRs.
- Major cleanup of loglet error handling, this removes the unnecessary leakage of internal loglet error types to bifrost. - No more superflous `Arc<>` of non-Clone errors. - FindTail on loglets cannot return Sealed error (enforcement via type-system)
This also adds an optional limit to loglet readstreams to enable creating a readstream with a pre-determined end offset.
In preparation to allow multi-segment chains, local loglets in single-node bootstrap now get unique random ids.
In preparation for the bifrost read_opt to be removed and replaced with `read()`
- Segments return the known tail_lsn which gets propagated to downstream components. - Renaming `read_opt()` to replace `read()` and making the non-blocking `read()` the de-facto way to read individual records. Readstream will be the only way for tailing readers. - Implements the intricate handling of `read()` during in-flight reconfiguration. This now requires bifrost to determine a safe known tail before allowing the `read()` to take place. - Testing read single record cross segments
…pdates ~20% P100 latency improvement when CPU bound according to bifrost-benchpress. Why didn't I do this from the get-go is beyond me! It'd be best to move away from the Vec<> structure to a fixed size struct to avoid searching but it'd break backward compatibility.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[Bifrost] Improve merge operator performance by coalescing metadata updates
~20% P100 latency improvement when CPU bound according to bifrost-benchpress. Why didn't I do this from the get-go is beyond me!
It'd be best to move away from the Vec<> structure to a fixed size struct to avoid searching but it'd break backward compatibility.
Stack created with Sapling. Best reviewed with ReviewStack.