-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-12218] Fixes ORC conjunction predicate push down #10377
[SPARK-12218] Fixes ORC conjunction predicate push down #10377
Conversation
Test build #48001 has finished for PR 10377 at commit
|
Test build #48005 has finished for PR 10377 at commit
|
Test build #48006 has finished for PR 10377 at commit
|
sorry.. My commit introduced a conflict... |
fe1ab88
to
e5913d2
Compare
@yhuai Nvm, rebased. |
Test build #48042 has finished for PR 10377 at commit
|
* child is inconvertible. Alas, `b.startAnd()` call can't be rolled back, and `b` is inconsistent | ||
* now. | ||
* | ||
* The workaround employed here is that, for `And`/`Or`/`Not`, we first try to convert their |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to mention for And Or Not
? createFilter
is dealing with the top level filters (they will be connected by AND
), right? I think it is important to emphasize that createFilter
is for top level filters.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
buildSearchArgument
is recursive, so nested And
/Or
/Not
within top level filters are also covered.
Test build #48270 has finished for PR 10377 at commit
|
f877ffa
to
6c59503
Compare
Test build #48344 has finished for PR 10377 at commit
|
LGTM. Merging to master. |
Thanks for the review! |
This PR is a follow-up of PR #10362.
Two major changes:
PR #10362 stops converting an
AND
predicate if any branch is inconvertible. On the other hand,OrcFilters
combines all filters into a single big conjunction first and then tries to convert it into ORCSearchArgument
. This means, if any filter is inconvertible, no filters can be pushed down. This PR fixes this issue by finding out all convertible filters first before doing the actual conversion.The reason behind the current implementation is mostly due to the limitation of ORC
SearchArgument
builder, which is documented in this PR in detail.2. Copied the
AND
predicate fix for ORC from #10362 to avoid merge conflict.Same as #10362, this PR targets master (2.0.0-SNAPSHOT), branch-1.6, and branch-1.5.