Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-19851][SQL] Add support for EVERY and ANY (SOME) aggregates #22797

Closed
wants to merge 2 commits into from

Conversation

dilipbiswal
Copy link
Contributor

@dilipbiswal dilipbiswal commented Oct 22, 2018

What changes were proposed in this pull request?

Implements Every, Some, Any aggregates in SQL. Logically these aggregate expressions are mapped to Min and Max, respectively.

Every(x) => Min(x)  where x is boolean.
Some(x) => Max(x) where x is boolean.

Any is a synonym for Some.

How was this patch tested?

Added tests in SQLQueryTestSuite, DataframeAggregateSuite

@SparkQA
Copy link

SparkQA commented Oct 23, 2018

Test build #97876 has finished for PR 22797 at commit 905652a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dilipbiswal
Copy link
Contributor Author

cc @cloud-fan @gatorsmile

@cloud-fan
Copy link
Contributor

Hi @dilipbiswal sorry for the back and forth, can you try one more approach? Basically we want to evaluate how the simplest logical rewrite looks like.

We can create unevaluatable EVERY and ANY expressions, and create a rule to replace them with MAX and MIN. This rule can be put in the beginning of optimizer like RewriteDistinctAggregates. We don't need to create a completed framework to rewrite agg functions.

@dilipbiswal
Copy link
Contributor Author

dilipbiswal commented Oct 23, 2018

@cloud-fan Sure.. Let me give that a try.
One question : So this would be a very specific rewrite like `RewriteEveryAnySome" as opposed to a generic rule to rewrite any aggregate expressions, right ? The reason i ask is because, i have tried the generic rewrite in https://github.com/dilipbiswal/spark/tree/SPARK-19851-rewrite

@cloud-fan
Copy link
Contributor

yea, just a special rule instead of a general agg function rewrite framework.

@dilipbiswal
Copy link
Contributor Author

@cloud-fan OK.. thanks a LOT.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants