Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PERF] Optimize filter on nested growables #1349

Merged
merged 7 commits into from
Sep 6, 2023
Merged

Conversation

jaychia
Copy link
Contributor

@jaychia jaychia commented Sep 6, 2023

This PR optimizes the new filter on nested arrays that uses growables

  1. Adds some fixes for FullNull impls of the arrays, which was broken due to some naming problems
  2. Adds microbenchmarks for filter
  3. Adds optimizations for our filter and growable impls:
    a. Make self.growable_validity optional, which speeds up the case where use_validity=False
    b. Adds an early return for when the mask is all true/all false
  4. Share a generic filter implementation across ListArray, StructArray and FixedSizeListArray

This brings us to parity within +-2% of our previous implementation of filter

@codecov
Copy link

codecov bot commented Sep 6, 2023

Codecov Report

Merging #1349 (1e3b7dc) into main (13d71e4) will decrease coverage by 0.02%.
The diff coverage is n/a.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1349      +/-   ##
==========================================
- Coverage   87.30%   87.28%   -0.02%     
==========================================
  Files          61       61              
  Lines        6000     6000              
==========================================
- Hits         5238     5237       -1     
- Misses        762      763       +1     

see 1 file with indirect coverage changes

@jaychia jaychia merged commit ccff1f4 into main Sep 6, 2023
27 checks passed
@jaychia jaychia deleted the jay/optimize-list-growable branch September 6, 2023 19:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant