Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filter mask #1327

Merged
merged 9 commits into from
Nov 18, 2024
Merged

Filter mask #1327

merged 9 commits into from
Nov 18, 2024

Conversation

gatesn
Copy link
Contributor

@gatesn gatesn commented Nov 15, 2024

Add a memoized filter mask.

FLUPs:

  • Swap to a single iter() function that returns an Enum of slices or indices based on selectivity. This forces all consumers to support both.
  • Actually... we should base it on avg_run_length. We can estimate run_count by performing a popcnt over each of the u64 slices of bits in the boolean buffer.

@@ -89,6 +90,18 @@ impl Canonical {
}
}

impl Canonical {
// Create an empty canonical array of the given dtype.
pub fn empty(dtype: &DType) -> VortexResult<Canonical> {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixes #1324

@gatesn gatesn enabled auto-merge (squash) November 16, 2024 20:51
Copy link
Member

@robert3005 robert3005 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we want to use once_cell instead of nightly feature. I think to reap the most benefits out of this you want to have RowMask hold FilterMask and not construct filtermask on demand

@@ -1,3 +1,4 @@
#![feature(once_cell_try)]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in other places we use the once_cell crate to get this feature without requiring one more nightly flag.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah let's chat Monday, there might be some convergence. I also want to make FilterMask internally memoize the indices or slices, instead of necessarily a boolean buffer.

@gatesn gatesn merged commit dbbfd56 into develop Nov 18, 2024
12 checks passed
@gatesn gatesn deleted the ngates/filter-mask branch November 18, 2024 11:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants