[PERF]: better locking of uncommitted tracking maps (decrease compaction time by 3x) #2736
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Reviewer ChecklistPlease leverage this checklist to ensure your code review is thorough before approving Testing, Bugs, Errors, Logs, Documentation
System Compatibility
Quality
|
codetheweb
commented
Aug 29, 2024
codetheweb
commented
Aug 29, 2024
rust/index/src/fulltext/types.rs
Outdated
| @@ -92,60 +93,71 @@ impl<'me> FullTextIndexWriter<'me> { | |||
|
|
|||
| async fn populate_frequencies_and_posting_lists_from_previous_version( | |||
| &self, | |||
| token: &str, | |||
| tokens: &Vec<Token>, | |||
There was a problem hiding this comment.
this is slightly weird, but the tantivy token type is already leaking out of our abstraction and this avoids having to remap the tokens to strs
not strongly opinionated here, ok going the remapping route
HammadB
reviewed
Aug 29, 2024
1 task
sanketkedia
reviewed
Aug 29, 2024
sanketkedia
reviewed
Aug 29, 2024
codetheweb
commented
Aug 29, 2024
| tracing::error!( | ||
| "Error populating frequencies and posting lists from previous version" | ||
| ); | ||
| return Err(FullTextIndexError::InvariantViolation); |
There was a problem hiding this comment.
I can add an extra Vec to track seen tokens in the above loop so we can retain this error if it's important
There was a problem hiding this comment.
I think it is fine, not worth the extra overhead
sanketkedia
reviewed
Aug 29, 2024
sanketkedia
approved these changes
Aug 29, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes
Instead of acquiring 2 locks for every token, we now acquire the locks at the document level. This improves compaction time by up to 3x.
I ran an experiment where I:
The partition size was 10,000.
Results (also includes changes from #2729 up to 14a744b):
Before:
After:
Here's a trace file for the two runs: locking-changes.trace.zip.
(Run 1 is after these changes, run 2 is before these changes.)
Test plan
How are these changes tested?
covered by existing tests
Documentation Changes
Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs repository?
n/a