Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEAT] any_value groupby aggregation #1941

Merged
merged 8 commits into from
Feb 29, 2024
Merged

[FEAT] any_value groupby aggregation #1941

merged 8 commits into from
Feb 29, 2024

Conversation

kevinzwang
Copy link
Member

@kevinzwang kevinzwang commented Feb 21, 2024

This function is parameterized by ignore_nulls, which attempts to find a non-null value in each group when true. However, usage of this parameter in the aggregation function would require some changes to DataFrame._agg() that I am going to save for later, since these changes will probably not be needed anymore once global expressions can be passed into GroupBy operations

Also in this PR: fixes to the count aggregation function

@kevinzwang kevinzwang linked an issue Feb 21, 2024 that may be closed by this pull request
@kevinzwang kevinzwang self-assigned this Feb 21, 2024
@github-actions github-actions bot added the enhancement New feature or request label Feb 21, 2024
Copy link

codecov bot commented Feb 21, 2024

Codecov Report

Attention: Patch coverage is 80.00000% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 83.93%. Comparing base (9c66a5e) to head (c5bfb4f).
Report is 10 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1941      +/-   ##
==========================================
- Coverage   85.40%   83.93%   -1.47%     
==========================================
  Files          55       55              
  Lines        6221     6132      -89     
==========================================
- Hits         5313     5147     -166     
- Misses        908      985      +77     
Files Coverage Δ
daft/expressions/expressions.py 91.39% <100.00%> (+0.41%) ⬆️
daft/logical/builder.py 89.16% <ø> (-1.00%) ⬇️
daft/dataframe/dataframe.py 87.15% <50.00%> (-0.13%) ⬇️

... and 15 files with indirect coverage changes

@kevinzwang kevinzwang marked this pull request as ready for review February 23, 2024 01:25
daft/dataframe/dataframe.py Outdated Show resolved Hide resolved
src/daft-core/src/array/ops/any_value.rs Outdated Show resolved Hide resolved
src/daft-core/src/array/ops/any_value.rs Outdated Show resolved Hide resolved
src/daft-core/src/array/ops/any_value.rs Outdated Show resolved Hide resolved
@@ -97,6 +97,14 @@ impl Series {
self.inner.max(groups)
}

pub fn any_value(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Zooming out a bit, we shouldn't have to implement this for each for the Arrays separately since this operation only has to work on the validity mask.
You could actually implement it on the series directly.

let mask = self.validity();
let indices = get_idx_from_bitmap(mask, groups);
self.take(indices)

I think count could also be refactored that way.
What do you think?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I implemented this idea for any_value, but the array-level count implementation is used directly in parts of the code (eg. for computing mean) so I think it's still useful to consolidate the logic there

@kevinzwang kevinzwang merged commit 73a71e6 into main Feb 29, 2024
27 of 29 checks passed
@kevinzwang kevinzwang deleted the kevin/any-value-agg branch February 29, 2024 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

any_value aggregation function
2 participants