[SQL] Improve SparkSQL Aggregates

[marmbrus]

• Add native min/max (was using hive before).
• Handle nulls correctly in Avg and Sum.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[rxin]

this is unrelated to this pr - but I just realized the way we are storing the aggregation buffer in Spark SQL uses much more memory than needed, because there are two extra pointers to expr/base, which is identical for every tuple.

[marmbrus]

Good point, though this is not an issue in the code gen version.
On May 7, 2014 2:28 PM, "Reynold Xin" [email protected] wrote:

In
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregates.scala:

@@ -86,6 +86,67 @@ abstract class AggregateFunction
override def newInstance() = makeCopy(productIterator.map { case a: AnyRef => a }.toArray)
}

+case class Min(child: Expression) extends PartialAggregate with trees.UnaryNode[Expression] {

• override def references = child.references
• override def nullable = child.nullable
• override def dataType = child.dataType
• override def toString = s"MIN($child)"

• override def asPartial: SplitEvaluation = {
• val partialMin = Alias(Min(child), "PartialMin")()
• SplitEvaluation(Min(partialMin.toAttribute), partialMin :: Nil)
• }

• override def newInstance() = new MinFunction(child, this)
  +}

+case class MinFunction(expr: Expression, base: AggregateExpression) extends AggregateFunction {

this is unrelated to this pr - but I just realized the way we are storing
the aggregation buffer in Spark SQL uses much more memory than needed,
because there are two extra pointers to expr/base, which is identical for
every tuple.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/683/files#r12404003
.

[rxin]

LGTM

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14784/

[marmbrus]

@pwendell, this should probably go in 1.0.

[rxin]

Merged.