Fix merge conflicts in coarse multuple executors #6

hermansc · 2015-10-24T14:34:18Z

Trying again... I think I merged the wrong way around last time.

Fixed conflicts so it can be merged with master. Let me know if I messed up. The conflict was in core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/CoarseMesosSchedulerBackend.scala and the executorTerminated function.

…oarse grain mode.

…ansform Clause #### What changes were proposed in this pull request? ```SQL FROM (FROM test SELECT TRANSFORM(key, value) USING 'cat' AS (`thing1` int, thing2 string)) t SELECT thing1 + 1 ``` This query returns an analysis error, like: ``` Failed to analyze query: org.apache.spark.sql.AnalysisException: cannot resolve '`thing1`' given input columns: [`thing1`, thing2]; line 3 pos 7 'Project [unresolvedalias(('thing1 + 1), None)] +- SubqueryAlias t +- ScriptTransformation [key#2,value#3], cat, [`thing1`#6,thing2#7], HiveScriptIOSchema(List(),List(),Some(org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe),Some(org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe),List((field.delim, )),List((field.delim, )),Some(org.apache.hadoop.hive.ql.exec.TextRecordReader),Some(org.apache.hadoop.hive.ql.exec.TextRecordWriter),false) +- SubqueryAlias test +- Project [_1#0 AS key#2,_2#1 AS value#3] +- LocalRelation [_1#0,_2#1], [[1,1],[2,2],[3,3],[4,4],[5,5]] ``` The backpacks of \`thing1\` should be cleaned before entering Parser/Analyzer. This PR fixes this issue. #### How was this patch tested? Added a test case and modified an existing test case Author: gatorsmile <[email protected]> Closes apache#11415 from gatorsmile/scriptTransform.

## What changes were proposed in this pull request? This PR aims to optimize GroupExpressions by removing repeating expressions. `RemoveRepetitionFromGroupExpressions` is added. **Before** ```scala scala> sql("select a+1 from values 1,2 T(a) group by a+1, 1+a, A+1, 1+A").explain() == Physical Plan == WholeStageCodegen : +- TungstenAggregate(key=[(a#0 + 1)#6,(1 + a#0)apache#7,(A#0 + 1)apache#8,(1 + A#0)apache#9], functions=[], output=[(a + 1)#5]) : +- INPUT +- Exchange hashpartitioning((a#0 + 1)#6, (1 + a#0)apache#7, (A#0 + 1)apache#8, (1 + A#0)apache#9, 200), None +- WholeStageCodegen : +- TungstenAggregate(key=[(a#0 + 1) AS (a#0 + 1)#6,(1 + a#0) AS (1 + a#0)apache#7,(A#0 + 1) AS (A#0 + 1)apache#8,(1 + A#0) AS (1 + A#0)apache#9], functions=[], output=[(a#0 + 1)#6,(1 + a#0)apache#7,(A#0 + 1)apache#8,(1 + A#0)apache#9]) : +- INPUT +- LocalTableScan [a#0], [[1],[2]] ``` **After** ```scala scala> sql("select a+1 from values 1,2 T(a) group by a+1, 1+a, A+1, 1+A").explain() == Physical Plan == WholeStageCodegen : +- TungstenAggregate(key=[(a#0 + 1)#6], functions=[], output=[(a + 1)#5]) : +- INPUT +- Exchange hashpartitioning((a#0 + 1)#6, 200), None +- WholeStageCodegen : +- TungstenAggregate(key=[(a#0 + 1) AS (a#0 + 1)#6], functions=[], output=[(a#0 + 1)#6]) : +- INPUT +- LocalTableScan [a#0], [[1],[2]] ``` ## How was this patch tested? Pass the Jenkins tests (with a new testcase) Author: Dongjoon Hyun <[email protected]> Closes apache#12590 from dongjoon-hyun/SPARK-14830.

tnachen and others added 5 commits September 1, 2015 20:05

[SPARK-5095] Support capping cores and launch mulitple executors in c…

42d59fd

…oarse grain mode.

Add documentation

47d2132

updates the test to comply to the updated TaskDescription.

0910004

Fix review comments.

fd0267f

Fix merge conflicts in tnacen coarse multiple executors

00bde2f

tnachen force-pushed the coarse_multiple_executors branch from fd0267f to 3dc1383 Compare October 30, 2015 22:36

tnachen force-pushed the coarse_multiple_executors branch 2 times, most recently from 08723f8 to 8a7a735 Compare November 21, 2015 06:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix merge conflicts in coarse multuple executors #6

Fix merge conflicts in coarse multuple executors #6

hermansc commented Oct 24, 2015

Fix merge conflicts in coarse multuple executors #6

Are you sure you want to change the base?

Fix merge conflicts in coarse multuple executors #6

Conversation

hermansc commented Oct 24, 2015