Implement an unify key-value iterator for Kvrocks #2004
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
git-hulk
force-pushed
the
add-new-key-iterator
branch
from
bfa1d0d
to
d7e34de
Compare
Currently, we need to iterate all keys in the database in different places like
the cluster migration and kvrocks2redis, but don't have an iterator for this purpose.
It's very error-prone to implement this in different places since Kvrocks may add
a new column family in the future, and we must be careful to iterate all keys in all column families.
This would be a burden for maintenance, So we want to implement an iterator for iterating keys.
```C++
DBIter iter(storage, read_option);
for (iter.Seek(); iter.Valid(); iter.Next()) {
if (iter.Type() == kRedisString || iter.Type() == kRedisJSON) {
// the string/json type didn't have subkeys
continue;
}
auto subkey_iter = iter.GetSubKeyIterator();
for (subkey_iter.Seek(); subkey_iter.Valid(); subkey_iter.Next()) {
// handle its subkey and value here
}
}
```
When using this iterator, it will iterate the metadata column family first and check its type,
if it's not a string or json, then it will iterate the corresponding column family to get subkeys.
That said, if we have a key foo with type hash, then the iterator will iterate foo and foo:field1, foo:field2, and so on.
This solution can bring those benefits:
- The codes look more intutive
- Can reuse this iterator if we want to iterate keys only
git-hulk
force-pushed
the
add-new-key-iterator
branch
from
d7e34de
to
e892901
Compare
caipengbo
reviewed
Jan 11, 2024
caipengbo
previously approved these changes
Jan 11, 2024
|
I have 2 question:
|
|
@jihuayu Thanks for your review.
I'm not quite sure about how to add test cases for bloom. Welcome to send a PR after this if anyone knows that. I will also add a test case for JSON and SortedInt.
Yes, it's not necessary to scan the score column family since it has enough information from the subkey column family. |
mapleFU
reviewed
Jan 12, 2024
caipengbo
approved these changes
Jan 12, 2024
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, we need to iterate all keys in the database in different places like the cluster migration and kvrocks2redis, but don't have an iterator for this purpose. It's very error-prone to implement this in different places since Kvrocks may add a new column family in the future, and we must be careful to iterate all keys in all column families. This would be a burden for maintenance, So we want to implement an iterator for iterating keys.
When using this iterator, it will iterate the metadata column family first and check its type, if it's not a string or JSON, then it will iterate the corresponding column family to get subkeys. That said, if we have a key foo with type hash, then the iterator will iterate foo and foo:field1, foo:field2, and so on.
This solution can bring those benefits:
This closes #1989