Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parquet pruneing incorrectly handles IS NOT NULL predicate #10049

Closed
Ted-Jiang opened this issue Apr 12, 2024 · 2 comments · Fixed by #10051
Closed

Parquet pruneing incorrectly handles IS NOT NULL predicate #10049

Ted-Jiang opened this issue Apr 12, 2024 · 2 comments · Fixed by #10051
Assignees
Labels
bug Something isn't working regression Something that used to work no longer does

Comments

@Ted-Jiang
Copy link
Member

          > Thanks @Ted-Jiang -- the support for `IS NOT NULL` is also quite neat.

@alamb sorry there is a bug here i will fix this it should

/// If set `with_not` to true: which means is not null
/// datafusion use false flag to prune unit (row group, page ..)
/// Given an expression reference to `expr`, if `expr` is a column expression,
/// returns a pruning expression in terms of IsNotNull that will evaluate to true
/// if the column may contain any nonnull values, and false if definitely does not contain
/// nonnull values which any null values.

Originally posted by @Ted-Jiang in #9989 (comment)

@Ted-Jiang
Copy link
Member Author

I will give a fix soon

@Ted-Jiang Ted-Jiang self-assigned this Apr 12, 2024
@alamb alamb added bug Something isn't working regression Something that used to work no longer does labels Apr 12, 2024
@alamb alamb changed the title Fix parquet prune handle IS NOT NULL Parquet pruneing incorrectly handles IS NOT NULL predicate Apr 12, 2024
@alamb
Copy link
Contributor

alamb commented Apr 12, 2024

Note this bug was not in any release of DataFusion yet

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working regression Something that used to work no longer does
Projects
None yet
2 participants