Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize isnan (2-5x faster) #12889

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

simonvandel
Copy link
Contributor

@simonvandel simonvandel commented Oct 12, 2024

Which issue does this PR close?

Closes #.

Rationale for this change

Apply same idea from #12881 to isnan.

What changes are included in this PR?

  • Add bench for is_nan
  • Optimize implementation

Are these changes tested?

Yes, with existing tests.

Are there any user-facing changes?

Yes, faster is_nan function.

isnan f32 array: 1024   time:   [794.46 ns 795.95 ns 797.51 ns]
                        change: [-60.752% -60.453% -60.230%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high mild

isnan f64 array: 1024   time:   [694.70 ns 698.25 ns 702.33 ns]
                        change: [-66.246% -66.029% -65.767%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
  6 (6.00%) high mild
  4 (4.00%) high severe

isnan f32 array: 4096   time:   [2.5207 µs 2.5282 µs 2.5358 µs]
                        change: [-64.245% -64.126% -64.015%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high mild

isnan f64 array: 4096   time:   [2.0779 µs 2.0812 µs 2.0847 µs]
                        change: [-74.140% -74.042% -73.950%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe

isnan f32 array: 8192   time:   [4.9547 µs 4.9730 µs 4.9971 µs]
                        change: [-74.219% -74.059% -73.933%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) high mild
  2 (2.00%) high severe

isnan f64 array: 8192   time:   [4.0478 µs 4.0562 µs 4.0652 µs]
                        change: [-82.993% -82.947% -82.900%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe

@github-actions github-actions bot added physical-expr Physical Expressions functions labels Oct 12, 2024
.collect::<$RETURN_TYPE>()
}};
}
use datafusion_common::Result;

/// Isnan SQL function
pub fn isnan(args: &[ArrayRef]) -> Result<ArrayRef> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this function is unused, and the real one is in datafusion/functions/src/math/nans.rs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are absolutely right.
Changing the right function does indeed give a speed-up.

I have left out the cleanup of the isnan of math_expressions.rs out of this PR now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #12917 to remove math_expressions.rs

@github-actions github-actions bot removed the physical-expr Physical Expressions label Oct 12, 2024
@simonvandel simonvandel changed the title Cleanup isnan using BooleanArray::from_unary Optimize isnan (2-5x faster) Oct 12, 2024
Copy link
Member

@jonahgao jonahgao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @simonvandel

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants