[SPARK-33183][SQL] Fix Optimizer rule EliminateSorts and add a physical rule to remove redundant sorts #30093

allisonwang-db · 2020-10-19T18:30:24Z

What changes were proposed in this pull request?

This PR aims to fix a correctness bug in the optimizer rule EliminateSorts. It also adds a new physical rule to remove redundant sorts that cannot be eliminated in the Optimizer rule after the bugfix.

Why are the changes needed?

A global sort should not be eliminated even if its child is ordered since we don't know if its child ordering is global or local. For example, in the following scenario, the first sort shouldn't be removed because it has a stronger guarantee than the second sort even if the sort orders are the same for both sorts.

Sort(orders, global = True, ...)
  Sort(orders, global = False, ...)

Since there is no straightforward way to identify whether a node's output ordering is local or global, we should not remove a global sort even if its child is already ordered.

Does this PR introduce any user-facing change?

Yes

How was this patch tested?

Unit tests

SparkQA · 2020-10-19T19:19:14Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34618/

SparkQA · 2020-10-19T19:48:40Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34618/

dongjoon-hyun · 2020-10-19T20:32:05Z

cc @viirya and @sunchao

SparkQA · 2020-10-19T23:20:54Z

Test build #130011 has finished for PR 30093 at commit bf6e45a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2020-10-20T01:46:34Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

@@ -1056,8 +1058,14 @@ object EliminateSorts extends Rule[LogicalPlan] {
    case s @ Sort(orders, _, child) if orders.isEmpty || orders.exists(_.child.foldable) =>
      val newOrders = orders.filterNot(_.child.foldable)
      if (newOrders.isEmpty) child else s.copy(order = newOrders)
-    case Sort(orders, true, child) if SortOrder.orderingSatisfies(child.outputOrdering, orders) =>


This was added in 2.4.

cc @cloud-fan

viirya · 2020-10-20T01:52:03Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

+        if SortOrder.orderingSatisfies(child.outputOrdering, orders) =>
+      (global, child) match {
+        case (false, _) => child
+        case (true, r: Range) => r


This assumes we know Range's global ordering in advance. This seems to leak physical stuff into the Optimizer.

maropu · 2020-10-20T07:45:04Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

+        case (false, _) => child
+        case (true, r: Range) => r
+        case (true, s @ Sort(_, true, _)) => s
+        case (true, _) => s.copy(child = recursiveRemoveSort(child))


Could you modify the code to handle this case in L1069?

maropu · 2020-10-20T07:46:12Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/EliminateSortsSuite.scala

    val optimized = Optimize.execute(unnecessaryReordered.analyze)
    val correctAnswer = orderedPlan.limit(2).select('a).analyze
    comparePlans(Optimize.execute(optimized), correctAnswer)
  }

+  test("should not remove global sort") {


nit: Add a prefix: SPARK-33183: for all the added tests?

tanelk · 2020-10-20T18:38:25Z

Shouldn't it be like this?

diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
index c37dac90c5..79610b4ab2 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
@@ -1058,15 +1058,9 @@ object EliminateSorts extends Rule[LogicalPlan] {
     case s @ Sort(orders, _, child) if orders.isEmpty || orders.exists(_.child.foldable) =>
       val newOrders = orders.filterNot(_.child.foldable)
       if (newOrders.isEmpty) child else s.copy(order = newOrders)
-    case s @ Sort(orders, global, child)
-        if SortOrder.orderingSatisfies(child.outputOrdering, orders) =>
-      (global, child) match {
-        case (false, _) => child
-        case (true, r: Range) => r
-        case (true, s @ Sort(_, true, _)) => s
-        case (true, _) => s.copy(child = recursiveRemoveSort(child))
-      }
-    case s @ Sort(_, _, child) => s.copy(child = recursiveRemoveSort(child))
+    case Sort(orders, false, child) if SortOrder.orderingSatisfies(child.outputOrdering, orders) =>
+      child
+    case s @ Sort(_, true, child) => s.copy(child = recursiveRemoveSort(child))
     case j @ Join(originLeft, originRight, _, cond, _) if cond.forall(_.deterministic) =>
       j.copy(left = recursiveRemoveSort(originLeft), right = recursiveRemoveSort(originRight))
     case g @ Aggregate(_, aggs, originChild) if isOrderIrrelevantAggs(aggs) =>

Two simpler rules:

Remove local sort if child satisfies the required ordering
Keep global sorts, but remove all sorts from child

This fails two tests:

Range test, but it is not correct because of what @viirya said
In the "remove two consecutive global sorts with same ordering" this keeps the parent sort, but in the test it keeps the child sort.

tanelk · 2020-10-20T18:59:50Z

Taking it a bit further we can also remove local child sorts inside a local sort:

diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
index c37dac90c5..dee3e40f0d 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
@@ -1058,33 +1058,29 @@ object EliminateSorts extends Rule[LogicalPlan] {
     case s @ Sort(orders, _, child) if orders.isEmpty || orders.exists(_.child.foldable) =>
       val newOrders = orders.filterNot(_.child.foldable)
       if (newOrders.isEmpty) child else s.copy(order = newOrders)
-    case s @ Sort(orders, global, child)
-        if SortOrder.orderingSatisfies(child.outputOrdering, orders) =>
-      (global, child) match {
-        case (false, _) => child
-        case (true, r: Range) => r
-        case (true, s @ Sort(_, true, _)) => s
-        case (true, _) => s.copy(child = recursiveRemoveSort(child))
-      }
-    case s @ Sort(_, _, child) => s.copy(child = recursiveRemoveSort(child))
+    case Sort(orders, false, child) if SortOrder.orderingSatisfies(child.outputOrdering, orders) =>
+      child
+    case s @ Sort(_, global, child) => s.copy(child = recursiveRemoveSort(child, global))
     case j @ Join(originLeft, originRight, _, cond, _) if cond.forall(_.deterministic) =>
       j.copy(left = recursiveRemoveSort(originLeft), right = recursiveRemoveSort(originRight))
     case g @ Aggregate(_, aggs, originChild) if isOrderIrrelevantAggs(aggs) =>
       g.copy(child = recursiveRemoveSort(originChild))
   }

-  private def recursiveRemoveSort(plan: LogicalPlan): LogicalPlan = plan match {
-    case Sort(_, _, child) => recursiveRemoveSort(child)
-    case other if canEliminateSort(other) =>
-      other.withNewChildren(other.children.map(recursiveRemoveSort))
+  private def recursiveRemoveSort(
+    plan: LogicalPlan, global: Boolean = true): LogicalPlan = plan match {
+    case Sort(_, false, child) if !global => recursiveRemoveSort(child, global)
+    case Sort(_, _, child) if global => recursiveRemoveSort(child, global)
+    case other if canEliminateSort(other, global) =>
+      other.withNewChildren(other.children.map(recursiveRemoveSort(_, global)))
     case _ => plan
   }

-  private def canEliminateSort(plan: LogicalPlan): Boolean = plan match {
+  private def canEliminateSort(plan: LogicalPlan, global: Boolean): Boolean = plan match {
     case p: Project => p.projectList.forall(_.deterministic)
     case f: Filter => f.condition.deterministic
-    case r: RepartitionByExpression => r.partitionExpressions.forall(_.deterministic)
-    case _: Repartition => true
+    case r: RepartitionByExpression => r.partitionExpressions.forall(_.deterministic) && global
+    case _: Repartition => global
     case _ => false
   }

allisonwang-db · 2020-10-20T21:23:08Z

I like the approach of using two simpler rules and removing the special case for the Range operator.

@tanelk I am not sure if I fully understand your proposed logic behind "remove local child sorts inside a local sort". Say we have this case

Sort('a.asc, false, _)
  Repartition
     Sort('b.asc, false, _)

With the proposed logic, the initial global value will be false, which will lead to the Repartition case to return false, and the recursiveRemoveSort will fail to eliminate the child local sort Sort('b.asc, false, _)

Can you provide an example when your logic can eliminate local child sorts inside a local sort while the current logic cannot?

SparkQA · 2020-10-20T21:58:13Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34665/

SparkQA · 2020-10-20T22:15:33Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34666/

SparkQA · 2020-10-20T22:20:54Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34665/

SparkQA · 2020-10-20T22:37:08Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34666/

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/EliminateSortsSuite.scala

maropu · 2020-10-20T23:40:50Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

@@ -1052,11 +1052,11 @@ object CombineFilters extends Rule[LogicalPlan] with PredicateHelper {
 *    function is order irrelevant
 */
 object EliminateSorts extends Rule[LogicalPlan] {
-  def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+  def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {


I remember this comment: #21072 (comment)

@allisonwang-db does it have to be transformUp?

Yes. This case is tricky. Let's consider this example: df.sortBy('a.asc).sortBy('a.asc).sortBy('a.asc)

Sort(a, false, _) // A Sort(a, false, _) // B Sort(a, false, _) // C ...

If we don't use transformUphere, the rule will keep both Sort B and C since it will not apply the rule to A's child (B) again. Plan after running this rule will be:

Sort(a, false, _) // B Sort(a, false, _) // C ...

And the Once strategy's idempotency will be broken since Sort B will be removed if we run this rule again.

tanelk · 2020-10-21T00:47:31Z

With the proposed logic, the initial global value will be false, which will lead to the Repartition case to return false, and the recursiveRemoveSort will fail to eliminate the child local sort Sort('b.asc, false, _)

Can you provide an example when your logic can eliminate local child sorts inside a local sort while the current logic cannot?

Ah yes, It seems we wouln't have to change the canEliminateSort at all.

SparkQA · 2020-10-21T01:32:29Z

Test build #130056 has finished for PR 30093 at commit 6d08848.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-10-21T01:52:28Z

Test build #130057 has finished for PR 30093 at commit 2966802.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-10-22T18:50:32Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34778/

SparkQA · 2020-10-22T19:12:26Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34778/

SparkQA · 2020-10-22T22:24:32Z

Test build #130172 has finished for PR 30093 at commit bcbd9fa.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class RemoveRedundantSorts(conf: SQLConf) extends Rule[SparkPlan]

cloud-fan · 2020-10-23T06:18:14Z

sql/core/src/test/scala/org/apache/spark/sql/execution/RemoveRedundantSortsSuite.scala

+  }
+
+  test("remove redundant sorts with sort merge join") {
+    withSQLConf(SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") {


can we use join hint to specify join type? it's more robust than tuning configs, especially when testing broadcast join.

cloud-fan · 2020-10-23T06:20:19Z

sql/core/src/test/scala/org/apache/spark/sql/execution/RemoveRedundantSortsSuite.scala

+      val innerJoinDesc1 = queryTemplate.format("JOIN", "t1.key DESC")
+      checkSorts(innerJoinDesc1, 1, 1)
+
+      val leftOuterJoinDesc = queryTemplate.format("LEFT JOIN", "t1.key DESC")


we don't need to use LEFT JOIN if we use join hint to specify which side to broadcast.

SparkQA · 2020-10-24T00:18:32Z

Test build #130219 has finished for PR 30093 at commit 2e5bc2c.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-10-24T01:01:23Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34819/

SparkQA · 2020-10-24T01:24:41Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34819/

cloud-fan · 2020-10-26T09:10:09Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

-  def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+  def apply(plan: LogicalPlan): LogicalPlan = plan transform applyLocally
+
+  val applyLocally: PartialFunction[LogicalPlan, LogicalPlan] = {


nit: private val

maropu · 2020-10-26T11:28:33Z

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

@@ -1242,6 +1242,13 @@ object SQLConf {
    .booleanConf
    .createWithDefault(true)

+  val REMOVE_REDUNDANT_SORTS_ENABLED = buildConf("spark.sql.execution.removeRedundantSorts")


We need this config? cc: @cloud-fan

If we want to backport this, it's safer to have a config for the new rule, in case it has bugs.

viirya · 2020-10-27T17:13:19Z

sql/core/src/main/scala/org/apache/spark/sql/execution/RemoveRedundantSorts.scala

+ * if the child satisfies the required distribution so that it is safe to remove not only a
+ * local sort but also a global sort when its child already satisfies required sort orders.
+ */
+case class RemoveRedundantSorts(conf: SQLConf) extends Rule[SparkPlan] {


Actually I think we should not backport this to 2.4. This is a new rule and looks like an improvement.

We should only backport the fix of EliminateSorts to 2.4.

But as it is late and this PR is close to merge and we have a config for it. I'm okay if others think it is fine.

We should only backport the fix of EliminateSorts to 2.4.

But, the partial backport can cause performance regressin in 2.4 ? (global sort -> global sort case) #30093 (comment)

Yes, however, this is inevitable for such bug fix and correctness bug fix precedes performance. Since the physical rule is well separated and has a config, it looks okay.

viirya · 2020-10-27T18:24:32Z

The current change is out of the PR title and description. Before merging this, can you update the title and description? Thanks.

SparkQA · 2020-10-27T18:30:19Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34937/

SparkQA · 2020-10-27T18:54:03Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34937/

SparkQA · 2020-10-27T22:24:12Z

Test build #130335 has finished for PR 30093 at commit cdc7dbe.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-10-28T05:51:45Z

thanks, merging to master!

cloud-fan · 2020-10-28T05:52:51Z

@viirya @maropu what do you think about backporting? Shall we backport without the new rule or with it?

viirya · 2020-10-28T06:18:54Z

I'm okay to backport it with new rule as it has config.

cloud-fan · 2020-10-28T06:26:36Z

@allisonwang-db can you help to create the backport PR? we need to change the config "since version" accordingly.

maropu · 2020-10-28T12:12:50Z

+1 to backport this. (I think it'd be better to avoid the performance regression of existing user's queries as much as possible.)

…al rule to remove redundant sorts This PR aims to fix a correctness bug in the optimizer rule `EliminateSorts`. It also adds a new physical rule to remove redundant sorts that cannot be eliminated in the Optimizer rule after the bugfix. A global sort should not be eliminated even if its child is ordered since we don't know if its child ordering is global or local. For example, in the following scenario, the first sort shouldn't be removed because it has a stronger guarantee than the second sort even if the sort orders are the same for both sorts. ``` Sort(orders, global = True, ...) Sort(orders, global = False, ...) ``` Since there is no straightforward way to identify whether a node's output ordering is local or global, we should not remove a global sort even if its child is already ordered. Yes Unit tests Closes apache#30093 from allisonwang-db/fix-sort. Authored-by: allisonwang-db <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit 9fb4536) Signed-off-by: allisonwang-db <[email protected]>

allisonwang-db · 2020-10-30T02:54:12Z

I will update the master branch config version to 2.4.8 once the backport PRs are merged.

…hysical rule to remove redundant sorts Backport #30093 for branch-2.4. ### What changes were proposed in this pull request? This PR aims to fix a correctness bug in the optimizer rule EliminateSorts. It also adds a new physical rule to remove redundant sorts that cannot be eliminated in the Optimizer rule after the bugfix. ### Why are the changes needed? A global sort should not be eliminated even if its child is ordered since we don't know if its child ordering is global or local. For example, in the following scenario, the first sort shouldn't be removed because it has a stronger guarantee than the second sort even if the sort orders are the same for both sorts. ``` Sort(orders, global = True, ...) Sort(orders, global = False, ...) ``` Since there is no straightforward way to identify whether a node's output ordering is local or global, we should not remove a global sort even if its child is already ordered. ### Does this PR introduce any user-facing change? Yes ### How was this patch tested? Unit tests Closes #30194 from allisonwang-db/SPARK-33183-branch-2.4. Authored-by: allisonwang-db <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

…hysical rule to remove redundant sorts Backport #30093 for branch-3.0. I've updated the configuration version to 2.4.8. ### What changes were proposed in this pull request? This PR aims to fix a correctness bug in the optimizer rule EliminateSorts. It also adds a new physical rule to remove redundant sorts that cannot be eliminated in the Optimizer rule after the bugfix. ### Why are the changes needed? A global sort should not be eliminated even if its child is ordered since we don't know if its child ordering is global or local. For example, in the following scenario, the first sort shouldn't be removed because it has a stronger guarantee than the second sort even if the sort orders are the same for both sorts. ``` Sort(orders, global = True, ...) Sort(orders, global = False, ...) ``` Since there is no straightforward way to identify whether a node's output ordering is local or global, we should not remove a global sort even if its child is already ordered. ### Does this PR introduce any user-facing change? Yes ### How was this patch tested? Unit tests Closes #30195 from allisonwang-db/SPARK-33183-branch-3.0. Authored-by: allisonwang-db <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>

… version ### What changes were proposed in this pull request? This PR is a follow up for #30093 to updates the config `spark.sql.execution.removeRedundantSorts` version to 2.4.8. ### Why are the changes needed? To update the rule version it has been backported to 2.4. #30194 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? N/A Closes #30420 from allisonwang-db/spark-33183-follow-up. Authored-by: allisonwang-db <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

viirya reviewed Oct 20, 2020

View reviewed changes

maropu reviewed Oct 20, 2020

View reviewed changes

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/EliminateSortsSuite.scala Show resolved Hide resolved

maropu reviewed Oct 20, 2020

View reviewed changes

cloud-fan reviewed Oct 23, 2020

View reviewed changes

cloud-fan reviewed Oct 26, 2020

View reviewed changes

maropu reviewed Oct 26, 2020

View reviewed changes

viirya reviewed Oct 27, 2020

View reviewed changes

allisonwang-db added 4 commits October 27, 2020 10:36

add physical rule

5997320

update

cec266f

address comments

290cb4d

update

cdc7dbe

allisonwang-db force-pushed the fix-sort branch from 59f9cd4 to cdc7dbe Compare October 27, 2020 17:40

allisonwang-db changed the title ~~[SPARK-33183][SQL] Fix EliminateSorts bug when removing global sorts~~ [SPARK-33183][SQL] Fix Optimizer rule EliminateSorts and add physical rule to remove redundant sorts Oct 27, 2020

allisonwang-db changed the title ~~[SPARK-33183][SQL] Fix Optimizer rule EliminateSorts and add physical rule to remove redundant sorts~~ [SPARK-33183][SQL] Fix Optimizer rule EliminateSorts and add a physical rule to remove redundant sorts Oct 27, 2020

cloud-fan closed this in 9fb4536 Oct 28, 2020

allisonwang-db mentioned this pull request Oct 30, 2020

[SPARK-33183][SQL][2.4] Fix Optimizer rule EliminateSorts and add a physical rule to remove redundant sorts #30194

Closed

allisonwang-db mentioned this pull request Oct 30, 2020

[SPARK-33183][SQL][3.0] Fix Optimizer rule EliminateSorts and add a physical rule to remove redundant sorts #30195

Closed

This was referenced Nov 13, 2020

[SPARK-33472][SQL] Adjust RemoveRedundantSorts rule order #30373

Closed

[SPARK-33183][SQL][FOLLOW-UP] Update rule RemoveRedundantSorts config version #30420

Closed

sunchao mentioned this pull request Jan 8, 2021

[SPARK-34026][SQL] Inject repartition and sort nodes to satisfy required distribution and ordering #31083

Closed

allisonwang-db deleted the fix-sort branch January 19, 2024 01:21

[SPARK-33183][SQL] Fix Optimizer rule EliminateSorts and add a physical rule to remove redundant sorts #30093

[SPARK-33183][SQL] Fix Optimizer rule EliminateSorts and add a physical rule to remove redundant sorts #30093

Conversation

allisonwang-db commented Oct 19, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Oct 19, 2020

SparkQA commented Oct 19, 2020

dongjoon-hyun commented Oct 19, 2020

SparkQA commented Oct 19, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tanelk commented Oct 20, 2020

tanelk commented Oct 20, 2020

allisonwang-db commented Oct 20, 2020

SparkQA commented Oct 20, 2020

SparkQA commented Oct 20, 2020

SparkQA commented Oct 20, 2020

SparkQA commented Oct 20, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tanelk commented Oct 21, 2020

SparkQA commented Oct 21, 2020

SparkQA commented Oct 21, 2020

SparkQA commented Oct 22, 2020

SparkQA commented Oct 22, 2020

SparkQA commented Oct 22, 2020

cloud-fan Oct 23, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Oct 24, 2020

SparkQA commented Oct 24, 2020

SparkQA commented Oct 24, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

viirya commented Oct 27, 2020

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

SparkQA commented Oct 27, 2020

cloud-fan commented Oct 28, 2020

cloud-fan commented Oct 28, 2020

viirya commented Oct 28, 2020

cloud-fan commented Oct 28, 2020

maropu commented Oct 28, 2020 • edited Loading

allisonwang-db commented Oct 30, 2020

allisonwang-db commented Oct 19, 2020 •

edited

Loading

cloud-fan Oct 23, 2020 •

edited

Loading

maropu commented Oct 28, 2020 •

edited

Loading