-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ENH] Metadata indices #1724
[ENH] Metadata indices #1724
Conversation
…/chroma into beggers/string-metadata
Reviewer ChecklistPlease leverage this checklist to ensure your code review is thorough before approving Testing, Bugs, Errors, Logs, Documentation
System Compatibility
Quality
|
Reviewers: I think there are more |
rust/worker/build.rs
Outdated
@@ -17,6 +17,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> { | |||
.flag("-DHAVE_CXX0X") | |||
.flag("-fpic") | |||
.flag("-ftree-vectorize") | |||
// .flag("-w") // Uncomment to suppress c++ warnings. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems worth having. Happy to remove.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's leave it out for now - I prefer the warnings cause pain until we have the correct solution to deal with them!
rust/worker/src/blockstore/mod.rs
Outdated
@@ -1,2 +1,5 @@ | |||
mod positional_posting_list_value; | |||
mod types; | |||
|
|||
pub use positional_posting_list_value::*; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is the correct way to export these but please correct if not.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thats fine if you want to rexeport - you can also do pub mod positional_posting_list_value
which exports the whole module under blockstore::positional_posting_list_value
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That seems like the correct way to export this but I still want to re-export types at top-level I think. Regardless, exporting positional_posting_list_value
should be done in the PR that needs it, so deleting from here (Graphite learning curve :D)
use std::hash::{Hash, Hasher}; | ||
|
||
#[derive(Debug, Error)] | ||
pub(crate) enum BlockfileError { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
} | ||
for (key, rbm) in &self.uncommitted_rbms { | ||
// TODO we shouldn't clone. | ||
self.blockfile.set(key.clone(), Value::RoaringBitmapValue(rbm.clone())); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you may be able to use https://doc.rust-lang.org/std/collections/hash_map/struct.Drain.html
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Exactly what I wanted, ty
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ty! Merging once CI passes.
} | ||
for (key, rbm) in &self.uncommitted_rbms { | ||
// TODO we shouldn't clone. | ||
self.blockfile.set(key.clone(), Value::RoaringBitmapValue(rbm.clone())); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Exactly what I wanted, ty
rust/worker/src/blockstore/mod.rs
Outdated
@@ -1,2 +1,5 @@ | |||
mod positional_posting_list_value; | |||
mod types; | |||
|
|||
pub use positional_posting_list_value::*; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That seems like the correct way to export this but I still want to re-export types at top-level I think. Regardless, exporting positional_posting_list_value
should be done in the PR that needs it, so deleting from here (Graphite learning curve :D)
## Description of changes *Summarize the changes made by this PR.* - New functionality - Blockfile-based metadata indexing. Currently uses a HashmapBlockfile, easy to change. ## Test plan *How are these changes tested?* - [x] Tests pass locally with `pytest` for python, `yarn test` for js ## Documentation Changes *Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs repository](https://github.com/chroma-core/docs)?* --------- Co-authored-by: hammadb <[email protected]>
Description of changes
Summarize the changes made by this PR.
Test plan
How are these changes tested?
pytest
for python,yarn test
for jsDocumentation Changes
Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs repository?