[SPARK-15214][SQL] Code-generation for Generate #13065

hvanhovell · 2016-05-11T22:38:27Z

What changes were proposed in this pull request?

This PR adds code generation to Generate. It supports two code paths:

General TraversableOnce based iteration. This used for regular Generator (code generation supporting) expressions. This code path expects the expression to return a TraversableOnce[InternalRow] and it will iterate over the returned collection. This PR adds code generation for the stack generator.
Specialized ArrayData/MapData based iteration. This is used for the explode, posexplode & inline functions and operates directly on the ArrayData/MapData result that the child of the generator returns.

Benchmarks

I have added some benchmarks and it seems we can create a nice speedup for explode:

Environment

Java HotSpot(TM) 64-Bit Server VM 1.8.0_92-b14 on Mac OS X 10.11.6
Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz

Explode Array

Before

generate explode array:                  Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
generate explode array wholestage off         7377 / 7607          2.3         439.7       1.0X
generate explode array wholestage on          6055 / 6086          2.8         360.9       1.2X

After

generate explode array:                  Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
generate explode array wholestage off         7432 / 7696          2.3         443.0       1.0X
generate explode array wholestage on           631 /  646         26.6          37.6      11.8X

Explode Map

Before

generate explode map:                    Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
generate explode map wholestage off         12792 / 12848          1.3         762.5       1.0X
generate explode map wholestage on          11181 / 11237          1.5         666.5       1.1X

After

generate explode map:                    Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
generate explode map wholestage off         10949 / 10972          1.5         652.6       1.0X
generate explode map wholestage on             870 /  913         19.3          51.9      12.6X

Posexplode

Before

generate posexplode array:               Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
generate posexplode array wholestage off      7547 / 7580          2.2         449.8       1.0X
generate posexplode array wholestage on       5786 / 5838          2.9         344.9       1.3X

After

generate posexplode array:               Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
generate posexplode array wholestage off      7535 / 7548          2.2         449.1       1.0X
generate posexplode array wholestage on        620 /  624         27.1          37.0      12.1X

Inline

Before

generate inline array:                   Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
generate inline array wholestage off          6935 / 6978          2.4         413.3       1.0X
generate inline array wholestage on           6360 / 6400          2.6         379.1       1.1X

After

generate inline array:                   Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
generate inline array wholestage off          6940 / 6966          2.4         413.6       1.0X
generate inline array wholestage on           1002 / 1012         16.7          59.7       6.9X

Stack

Before

generate stack:                          Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
generate stack wholestage off               12980 / 13104          1.3         773.7       1.0X
generate stack wholestage on                11566 / 11580          1.5         689.4       1.1X

After

generate stack:                          Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
generate stack wholestage off               12875 / 12949          1.3         767.4       1.0X
generate stack wholestage on                   840 /  845         20.0          50.0      15.3X

How was this patch tested?

Existing tests.

SparkQA · 2016-05-11T23:38:29Z

Test build #58418 has finished for PR 13065 at commit 5254559.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Explode(child: Expression) extends UnaryExpression with Generator

rxin · 2016-05-12T06:09:01Z

Looks like a good start. Once you flush this out more, can you print the generated code ("explain codegen ..."), and do some microbenchmarks similar to the ones done in the benchmark package?

hvanhovell · 2016-05-12T08:45:12Z

Seems like the build is stuck. I might have introduced a infinite loop :(...

SparkQA · 2016-05-12T10:37:35Z

Test build #58454 has finished for PR 13065 at commit 5d068b5.

This patch fails from timeout after a configured wait of 250m.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-12T18:33:35Z

Test build #58505 has finished for PR 13065 at commit b2d663b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-12T21:06:14Z

Test build #58513 has finished for PR 13065 at commit e04d66f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-12T23:42:56Z

Test build #58517 has finished for PR 13065 at commit 43a04bf.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-17T18:01:25Z

Test build #58705 has finished for PR 13065 at commit 7b4772d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class CodegenFallbackExpression(child: Expression)

SparkQA · 2016-06-03T23:52:03Z

Test build #59965 has finished for PR 13065 at commit 5cfba19.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-07T19:30:50Z

Test build #60131 has finished for PR 13065 at commit f86da0f.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2016-06-07T23:39:01Z

Test build #60137 has finished for PR 13065 at commit 60da24e.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MyTransformer(override val uid: String)
- trait DefaultParamsWritable extends MLWritable
- trait DefaultParamsReadable[T] extends MLReadable[T]
- case class ValidateExternalType(child: Expression, expected: DataType)
- implicit class AttributeSeq(val attrs: Seq[Attribute]) extends Serializable

# Conflicts: # sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala # sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala

SparkQA · 2016-11-04T01:33:37Z

Test build #68091 has finished for PR 13065 at commit 7b7fa6e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

davies · 2016-11-03T19:21:24Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala

+    ctx.addMutableState("InternalRow[]", rowData, s"this.$rowData = new InternalRow[$numRows];")
+    val values = children.tail
+    val dataTypes = values.take(numFields).map(_.dataType)
+    val rows = for (row <- 0 until numRows) yield {


Should we split these into multiple funcitons just in case of numRow is large?

davies · 2016-11-03T19:24:39Z

sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala

+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    // We need to add some code here for terminating generators.


Is this a TODO?

Added a comment to the doc. The problem is that we currently do not have a generator in Spark SQL that requires this.

davies · 2016-11-03T19:27:32Z

sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala

+    }
+
+    // Generate the driving expression.
+    val data = boundGenerator.genCode(ctx)


Will moving this into codeGenCollection/codeGenTraversableOnce be better?

davies · 2016-11-08T17:29:41Z

sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/MiscBenchmark.scala

+    generate explode array:                  Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
+    ------------------------------------------------------------------------------------------------
+    generate explode array wholestage off         6920 / 7129          2.4         412.5       1.0X
+    generate explode array wholestage on           623 /  646         26.9          37.1      11.1X


Could you post the result before and after this patch? so we can know what's the improvement come from this patch.

Post it in the PR description should be enough

SparkQA · 2016-11-15T22:32:33Z

Test build #68671 has finished for PR 13065 at commit f81eed7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

davies · 2016-11-17T18:43:44Z

The performance improvements look awesome, I will finished the review today.

davies · 2016-11-17T19:04:09Z

sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala

@@ -40,6 +42,10 @@ private[execution] sealed case class LazyIterator(func: () => TraversableOnce[In
 * output of each into a new stream of rows.  This operation is similar to a `flatMap` in functional
 * programming with one important additional feature, which allows the input rows to be joined with
 * their output.
+ *
+ * This operator supports whole stage code generation for generators that do not implement
+ * terminate().


Where do we check it and fallback?

Could be done in

override def supportCodegen(): Boolean = { generator.terminate().isEmpty }

I don't think this holds. This method implies that the generator holds some state, and that calling this on an un-used generator might return the correct result.

I have added an explicit GeneratorWithCodegen trait which all generators must implement if they want to be eligible for code generation. The terminate() method in this trait has been closed off.

I see, those Generator with terminate implements CodegenFallback, then GenerateExec will also fallback, nvm.

davies · 2016-11-17T19:08:42Z

sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala

+    val init = concatIfOuter(s"boolean $hasNextCode", s", $outerVal = true")
+    val check = concatIfOuter(hasNext, s"|| $outerVal")
+    val update = concatIfOuter(hasNextCode, s", $outerVal = false")
+    val next = if (outer) s"$hasNext ? $iterator.next() : null" else s"$iterator.next()"


minor: will these be easier to understand by having different branch for outer? for example:

if (outer) { } else { }

It will end with more lines of codes, but easier to understand.

Sorry that I meant put the whole block in it, avoid the using of cancatIfOuter

Ah ok, I'll update :)

davies · 2016-11-17T19:17:50Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala

+    ctx.addMutableState("InternalRow[]", rowData, s"this.$rowData = new InternalRow[$numRows];")
+    val values = children.tail
+    val dataTypes = values.take(numFields).map(_.dataType)
+    val code = ctx.splitExpressions(ctx.INPUT_ROW, Seq.tabulate(numRows) { row =>


Unfortunately, splitExpressions does not work with whole stage codegen, because the input of children expression is not ctx.INPUT_ROW, we have to fallback if there are many rows.

Can you add a test for that (many rows)?

I have added a test to check if it will fail compilation for a large stack expression. Testing the actual fallback is nearly impossible. I would need to change something on this line: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala#L359

davies · 2016-11-18T00:53:49Z

sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala

+    assert(ds.collect() === Array(Row(0, 1), Row(0, 2), Row(1, 2), Row(1, 3)))
+  }
+
+  test("large inline generate should fail in WholeStageCodegen") {


Should this still work (by fallback)?

Yeah it still falls back like it should (I tested that manually). The problem is that the fallback code path is not testable at the moment. So I am currently testing if inline fails for large inline cases, assuming that the fallback works.

I mean it should not reply on the fallback for failed compiling (whole plan will fallback), but don't use whole stage codegen for this Generate.

This could be done by checking it in Generate.supportCodegen()

SparkQA · 2016-11-18T01:48:08Z

Test build #68804 has finished for PR 13065 at commit 29c606a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

davies · 2016-11-18T22:54:47Z

sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala

+       |  InternalRow $current = (InternalRow)($next);
+       |  ${consume(ctx, input ++ values)}
+       |}
+     """.stripMargin


minor: I'm thinking of

if (outer) { s""" |${data.code} |scala.collection.Iterator<InternalRow> $iterator = ${data.value}.toIterator(); |boolean $outerVal = true; |while ($iterator.hasNext() || $outerVal) { | $numOutput.add(1); | InternalRow $current = (InternalRow)($iterator.hasNext()? $iterator.next() : null); | $outerVal = false; | ${consume(ctx, input ++ values)} |} """.stripMargin } else { s""" |${data.code} |scala.collection.Iterator<InternalRow> $iterator = ${data.value}.toIterator(); |while ($iterator.hasNext()) { | $numOutput.add(1); | InternalRow $current = (InternalRow)($iterator.next()); | ${consume(ctx, input ++ values)} |} """.stripMargin }

…r and regular code paths more explicit.

SparkQA · 2016-11-19T01:10:53Z

Test build #68875 has finished for PR 13065 at commit af9a516.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-11-19T02:24:12Z

Test build #68876 has finished for PR 13065 at commit 3146cc5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

kiszk · 2016-11-19T02:40:05Z

sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala

+    // iterator contains no input. We do this by adding an 'outer' variable which guarantees
+    // execution of the first iteration even if there is no input. Evaluation of the iterator is
+    // prevented by checks in the next() and accessor code.
+    val outerVal = ctx.freshName("outer")


minor comment: this statement can be move into a then clause.

SparkQA · 2016-11-19T13:51:45Z

Test build #68885 has finished for PR 13065 at commit ffd5ef8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell · 2016-11-19T13:53:38Z

retest this please

SparkQA · 2016-11-19T16:24:19Z

Test build #68891 has finished for PR 13065 at commit ffd5ef8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

davies · 2016-11-20T07:14:17Z

LGTM

rxin · 2016-11-20T07:54:04Z

Merging in master.

## What changes were proposed in this pull request? This PR adds code generation to `Generate`. It supports two code paths: - General `TraversableOnce` based iteration. This used for regular `Generator` (code generation supporting) expressions. This code path expects the expression to return a `TraversableOnce[InternalRow]` and it will iterate over the returned collection. This PR adds code generation for the `stack` generator. - Specialized `ArrayData/MapData` based iteration. This is used for the `explode`, `posexplode` & `inline` functions and operates directly on the `ArrayData`/`MapData` result that the child of the generator returns. ### Benchmarks I have added some benchmarks and it seems we can create a nice speedup for explode: #### Environment ``` Java HotSpot(TM) 64-Bit Server VM 1.8.0_92-b14 on Mac OS X 10.11.6 Intel(R) Core(TM) i7-4980HQ CPU 2.80GHz ``` #### Explode Array ##### Before ``` generate explode array: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------ generate explode array wholestage off 7377 / 7607 2.3 439.7 1.0X generate explode array wholestage on 6055 / 6086 2.8 360.9 1.2X ``` ##### After ``` generate explode array: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------ generate explode array wholestage off 7432 / 7696 2.3 443.0 1.0X generate explode array wholestage on 631 / 646 26.6 37.6 11.8X ``` #### Explode Map ##### Before ``` generate explode map: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------ generate explode map wholestage off 12792 / 12848 1.3 762.5 1.0X generate explode map wholestage on 11181 / 11237 1.5 666.5 1.1X ``` ##### After ``` generate explode map: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------ generate explode map wholestage off 10949 / 10972 1.5 652.6 1.0X generate explode map wholestage on 870 / 913 19.3 51.9 12.6X ``` #### Posexplode ##### Before ``` generate posexplode array: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------ generate posexplode array wholestage off 7547 / 7580 2.2 449.8 1.0X generate posexplode array wholestage on 5786 / 5838 2.9 344.9 1.3X ``` ##### After ``` generate posexplode array: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------ generate posexplode array wholestage off 7535 / 7548 2.2 449.1 1.0X generate posexplode array wholestage on 620 / 624 27.1 37.0 12.1X ``` #### Inline ##### Before ``` generate inline array: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------ generate inline array wholestage off 6935 / 6978 2.4 413.3 1.0X generate inline array wholestage on 6360 / 6400 2.6 379.1 1.1X ``` ##### After ``` generate inline array: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------ generate inline array wholestage off 6940 / 6966 2.4 413.6 1.0X generate inline array wholestage on 1002 / 1012 16.7 59.7 6.9X ``` #### Stack ##### Before ``` generate stack: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------ generate stack wholestage off 12980 / 13104 1.3 773.7 1.0X generate stack wholestage on 11566 / 11580 1.5 689.4 1.1X ``` ##### After ``` generate stack: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------ generate stack wholestage off 12875 / 12949 1.3 767.4 1.0X generate stack wholestage on 840 / 845 20.0 50.0 15.3X ``` ## How was this patch tested? Existing tests. Author: Herman van Hovell <[email protected]> Author: Herman van Hovell <[email protected]> Closes apache#13065 from hvanhovell/SPARK-15214.

This PR adds code generation to `Generate`. It supports two code paths: - General `TraversableOnce` based iteration. This used for regular `Generator` (code generation supporting) expressions. This code path expects the expression to return a `TraversableOnce[InternalRow]` and it will iterate over the returned collection. This PR adds code generation for the `stack` generator. - Specialized `ArrayData/MapData` based iteration. This is used for the `explode`, `posexplode` & `inline` functions and operates directly on the `ArrayData`/`MapData` result that the child of the generator returns. I have added some benchmarks and it seems we can create a nice speedup for explode: ``` Java HotSpot(TM) 64-Bit Server VM 1.8.0_92-b14 on Mac OS X 10.11.6 Intel(R) Core(TM) i7-4980HQ CPU 2.80GHz ``` ``` generate explode array: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------ generate explode array wholestage off 7377 / 7607 2.3 439.7 1.0X generate explode array wholestage on 6055 / 6086 2.8 360.9 1.2X ``` ``` generate explode array: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------ generate explode array wholestage off 7432 / 7696 2.3 443.0 1.0X generate explode array wholestage on 631 / 646 26.6 37.6 11.8X ``` ``` generate explode map: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------ generate explode map wholestage off 12792 / 12848 1.3 762.5 1.0X generate explode map wholestage on 11181 / 11237 1.5 666.5 1.1X ``` ``` generate explode map: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------ generate explode map wholestage off 10949 / 10972 1.5 652.6 1.0X generate explode map wholestage on 870 / 913 19.3 51.9 12.6X ``` ``` generate posexplode array: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------ generate posexplode array wholestage off 7547 / 7580 2.2 449.8 1.0X generate posexplode array wholestage on 5786 / 5838 2.9 344.9 1.3X ``` ``` generate posexplode array: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------ generate posexplode array wholestage off 7535 / 7548 2.2 449.1 1.0X generate posexplode array wholestage on 620 / 624 27.1 37.0 12.1X ``` ``` generate inline array: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------ generate inline array wholestage off 6935 / 6978 2.4 413.3 1.0X generate inline array wholestage on 6360 / 6400 2.6 379.1 1.1X ``` ``` generate inline array: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------ generate inline array wholestage off 6940 / 6966 2.4 413.6 1.0X generate inline array wholestage on 1002 / 1012 16.7 59.7 6.9X ``` ``` generate stack: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------ generate stack wholestage off 12980 / 13104 1.3 773.7 1.0X generate stack wholestage on 11566 / 11580 1.5 689.4 1.1X ``` ``` generate stack: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------ generate stack wholestage off 12875 / 12949 1.3 767.4 1.0X generate stack wholestage on 840 / 845 20.0 50.0 15.3X ``` Existing tests. Author: Herman van Hovell <[email protected]> Author: Herman van Hovell <[email protected]> Closes apache#13065 from hvanhovell/SPARK-15214.

hvanhovell added 2 commits May 12, 2016 00:27

CG for Generate/Explode

5254559

Merge remote-tracking branch 'apache-github/master' into SPARK-15214

9df56a9

Fix compilation and binding errors.

5d068b5

Fix infinite loop on continue.

b2d663b

Fix outer.

e04d66f

Really really fix lateral outer view explode(...).

43a04bf

Add benchmark & fix subexpressionsuite.

7b4772d

hvanhovell added 8 commits May 22, 2016 22:00

Merge remote-tracking branch 'apache-github/master' into SPARK-15214

b721b60

Generate WIP

09513e7

Update GenerateExec

f7c2307

Merge remote-tracking branch 'apache-github/master' into SPARK-15214

f5bd9cf

Merge remote-tracking branch 'apache-github/master' into SPARK-15214

dba4240

Further tweaking...

49f9e7f

Proper support for json_tuple.

b3531cb

Merge remote-tracking branch 'apache-github/master' into SPARK-15214

5cfba19

Use TraversableOnce for regular Generators.

f86da0f

hvanhovell changed the title ~~[SPARK-15214][SQL] Code-generation for Generate [WIP]~~ [SPARK-15214][SQL] Code-generation for Generate Jun 7, 2016

Merge remote-tracking branch 'apache-github/master' into SPARK-15214

60da24e

Merge remote-tracking branch 'apache-github/master' into SPARK-15214

87688b1

Merge remote-tracking branch 'apache-github/master' into SPARK-15214

7b7fa6e

# Conflicts: # sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala # sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala

davies reviewed Nov 9, 2016

View reviewed changes

hvanhovell added 2 commits November 15, 2016 03:45

Merge remote-tracking branch 'apache-github/master' into SPARK-15214

ebd9d8c

Code review

f81eed7

davies reviewed Nov 17, 2016

View reviewed changes

code review

29c606a

davies reviewed Nov 18, 2016

View reviewed changes

code review 2

af9a516

davies reviewed Nov 18, 2016

View reviewed changes

Add proper fallback for 'Stack' generator. Make traversable once oute…

3146cc5

…r and regular code paths more explicit.

kiszk reviewed Nov 19, 2016

View reviewed changes

Minor thing

ffd5ef8

asfgit closed this in 7ca7a63 Nov 20, 2016

[SPARK-15214][SQL] Code-generation for Generate #13065

[SPARK-15214][SQL] Code-generation for Generate #13065

Conversation

hvanhovell commented May 11, 2016 • edited Loading

What changes were proposed in this pull request?

Benchmarks

Environment

Explode Array

Before

After

Explode Map

Before

After

Posexplode

Before

After

Inline

Before

After

Stack

Before

After

How was this patch tested?

SparkQA commented May 11, 2016

rxin commented May 12, 2016

hvanhovell commented May 12, 2016

SparkQA commented May 12, 2016

SparkQA commented May 12, 2016

SparkQA commented May 12, 2016

SparkQA commented May 12, 2016

SparkQA commented May 17, 2016

SparkQA commented Jun 3, 2016

SparkQA commented Jun 7, 2016

SparkQA commented Jun 7, 2016

SparkQA commented Nov 4, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hvanhovell Nov 15, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Nov 15, 2016

davies commented Nov 17, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davies Nov 17, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davies Nov 18, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Nov 18, 2016

Choose a reason for hiding this comment

SparkQA commented Nov 19, 2016

SparkQA commented Nov 19, 2016

Choose a reason for hiding this comment

SparkQA commented Nov 19, 2016

hvanhovell commented Nov 19, 2016

SparkQA commented Nov 19, 2016

davies commented Nov 20, 2016

rxin commented Nov 20, 2016

hvanhovell commented May 11, 2016 •

edited

Loading

hvanhovell Nov 15, 2016 •

edited

Loading

davies Nov 17, 2016 •

edited

Loading

davies Nov 18, 2016 •

edited

Loading