-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-11677][SQL][FOLLOW-UP] Add tests for checking the ORC filter creation against pushed down filters. #10341
Conversation
As I talked with @liancheng, this PR is not covering But as I see Hive's latest codes, this is accessible. So, I think I can add some tests about |
Test build #47878 has finished for PR 10341 at commit
|
Hey @HyukjinKwon, actually I hit the same issue for testing Please refer to this file for details. |
@liancheng Thanks! I will try to apply that way. |
In this commit, I used string expression I am not too sure if generalising them with string template is appropriate though. I can't come up with a better idea for now. |
Oh, actually I didn't mean that you should use string comparison to test all ORC filters. It's perfectly OK to use The only reason that I suggested using string comparison here is that ORC doesn't expose necessary interfaces similar to |
Test build #48104 has finished for PR 10341 at commit
|
Ah. I will correct them soon! |
In the commits above, I added tests for logical operators separately. Although it does not check all the combinations across types with logical operators, I think this would be okay because the basic comparison operators are being tested across types. Could we add some tests for logical operators across types later if they are necessary in a way? I do not want to make the tests hacky clunky. |
Test build #48162 has finished for PR 10341 at commit
|
Test build #48164 has finished for PR 10341 at commit
|
Test build #48165 has finished for PR 10341 at commit
|
Thanks for working on this! Merging to master. |
Hit network issue and couldn't fetch from GitHub... Finally got it merged. |
https://issues.apache.org/jira/browse/SPARK-11677
Although it checks correctly the filters by the number of results if ORC filter-push-down is enabled, the filters themselves are not being tested.
So, this PR includes the test similarly with
ParquetFilterSuite
.Since the results are checked by
OrcQuerySuite
, thisOrcFilterSuite
only checks if the appropriate filters are created.One thing different with
ParquetFilterSuite
here is, it does not check the results because that is checked inOrcQuerySuite
.