Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

datafusion doesn't process predicate pushdown correctly when there is outer join #1586

Closed
xudong963 opened this issue Jan 16, 2022 · 3 comments · Fixed by #1618
Closed

datafusion doesn't process predicate pushdown correctly when there is outer join #1586

xudong963 opened this issue Jan 16, 2022 · 3 comments · Fixed by #1618
Labels
bug Something isn't working datafusion Changes in the datafusion crate

Comments

@xudong963
Copy link
Member

xudong963 commented Jan 16, 2022

Describe the bug

Now datafusion doesn't process predicate pushdown correctly when there is outer join. I have discussed much in #1339 with @alamb. I think we ended up with a consensus.

Notes that filtering conditions fall into two categories: join condition and where condition, I'll explain them separately in the following:

  1. join condition: if the filter condition contains the non-nullable-side table, it can't be pushed down.
  2. where condition: if the filter condition contains the nullable-side table, it can't be pushed down. But it's can be optimized, I describe how to process the case in convert outer join to inner join to improve performance #1585.

Additional context
related issues and pr: #1321 , #1585, #1339

@xudong963 xudong963 added the bug Something isn't working label Jan 16, 2022
@xudong963 xudong963 changed the title Predicate pushdown requires special process for outer Join. datafusion doesn't process predicate pushdown correctly when there is outer join Jan 16, 2022
@xudong963
Copy link
Member Author

cc @alamb

@james727
Copy link
Contributor

I can try this one if no one is working on it yet.

@xudong963
Copy link
Member Author

xudong963 commented Jan 17, 2022

I can try this one if no one is working on it yet.

I believe no one is working on it, please do it! Thanks, @james727 If you have any problems, please feel free to ping me!

@alamb alamb added the datafusion Changes in the datafusion crate label Feb 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working datafusion Changes in the datafusion crate
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants