[SPARK-3666] Extract interfaces for EdgeRDD and VertexRDD #2530

ankurdave · 2014-09-25T02:35:56Z

This discourages users from calling the VertexRDD and EdgeRDD constructor and makes it easier for future changes to ensure backward compatibility.

This breaks binary compatibility with the error java.lang.IncompatibleClassChangeError: Found interface org.apache.spark.graphx.VertexRDD, but class was expected The solution is to do a clean rebuild (sbt/sbt clean assembly).

SparkQA · 2014-09-25T02:39:26Z

QA tests have started for PR 2530 at commit 620e603.

This patch merges cleanly.

SparkQA · 2014-09-25T03:50:51Z

QA tests have finished for PR 2530 at commit 620e603.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait EdgeRDD[@specialized ED, VD] extends RDD[Edge[ED]]
- trait VertexRDD[@specialized VD] extends RDD[(VertexId, VD)]
- class EdgeRDDImpl[@specialized ED: ClassTag, VD: ClassTag](
- class VertexRDDImpl[@specialized VD](

AmplabJenkins · 2014-09-25T03:50:54Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20777/

rxin · 2014-09-27T05:57:14Z

graphx/src/main/scala/org/apache/spark/graphx/EdgeRDD.scala

-    }, preservesPartitioning = true))
-  }
-
+trait EdgeRDD[@specialized ED, VD] extends RDD[Edge[ED]] {


i think we should probably remove the specialized and test performance. maybe we can do that in a a separate pr

rxin · 2014-09-27T05:59:37Z

Looks good. I didn't look closely but I'm assuming you didn't change any impl logic. Let me know if that is not true.

SparkQA · 2014-10-09T22:40:14Z

QA tests have started for PR 2530 at commit 9ba4ec4.

This patch merges cleanly.

SparkQA · 2014-10-09T23:43:37Z

QA tests have finished for PR 2530 at commit 9ba4ec4.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait EdgeRDD[@specialized ED, VD] extends RDD[Edge[ED]]
- trait VertexRDD[@specialized VD] extends RDD[(VertexId, VD)]

AmplabJenkins · 2014-10-09T23:43:40Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21554/Test PASSed.

ankurdave · 2014-10-10T00:29:27Z

@rxin Right, this just contains refactoring and doc changes. I marked the Impl constructors package-private since they are accessed by the corresponding interface object (EdgeRDD.fromEdgePartitions and VertexRDD.apply). I'll evaluate specialization separately. PTAL

rxin · 2014-10-28T07:00:00Z

This LGTM.

One question related to this - should we use abstract class instead of trait? It would be better for maintaining binary compatibility across versions

SparkQA · 2014-11-01T22:29:50Z

QA tests have started for PR 2530 at commit 931b587.

This patch does not merge cleanly!

It makes no difference in benchmarks.

SparkQA · 2014-11-01T23:54:56Z

QA tests have started for PR 2530 at commit cbe15f2.

This patch does not merge cleanly!

Fixes a merge conflict created by ee29ef3. Conflicts: graphx/src/main/scala/org/apache/spark/graphx/EdgeRDD.scala

SparkQA · 2014-11-01T23:57:49Z

QA tests have finished for PR 2530 at commit 931b587.

This patch fails unit tests.
This patch does not merge cleanly!

AmplabJenkins · 2014-11-01T23:57:53Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22704/
Test FAILed.

SparkQA · 2014-11-02T00:05:05Z

Test build #22716 has started for PR 2530 at commit 24201d4.

This patch merges cleanly.

SparkQA · 2014-11-02T01:07:09Z

QA tests have finished for PR 2530 at commit cbe15f2.

This patch fails unit tests.
This patch does not merge cleanly!

AmplabJenkins · 2014-11-02T01:07:12Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22714/
Test FAILed.

SparkQA · 2014-11-02T01:21:29Z

Test build #22716 has finished for PR 2530 at commit 24201d4.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-02T01:21:32Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22716/
Test FAILed.

rxin · 2014-11-02T04:35:22Z

Jenkins, retest this please.

SparkQA · 2014-11-02T04:39:49Z

Test build #22741 has started for PR 2530 at commit 24201d4.

This patch merges cleanly.

SparkQA · 2014-11-02T05:57:46Z

Test build #22741 has finished for PR 2530 at commit 24201d4.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class EdgeRDD[ED, VD](
- abstract class VertexRDD[VD](

AmplabJenkins · 2014-11-02T05:57:49Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22741/
Test FAILed.

rxin · 2014-11-02T06:58:19Z

Jenkins, retest this please.

SparkQA · 2014-11-02T07:04:50Z

Test build #22754 has started for PR 2530 at commit 24201d4.

This patch merges cleanly.

SparkQA · 2014-11-02T09:25:15Z

Test build #22754 has finished for PR 2530 at commit 24201d4.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class EdgeRDD[ED, VD](
- abstract class VertexRDD[VD](

AmplabJenkins · 2014-11-02T09:25:18Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22754/
Test FAILed.

SparkQA · 2014-11-11T00:18:36Z

Test build #516 has started for PR 2530 at commit 24201d4.

This patch merges cleanly.

SparkQA · 2014-11-11T01:46:49Z

Test build #516 has finished for PR 2530 at commit 24201d4.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class EdgeRDD[ED, VD](
- abstract class VertexRDD[VD](

rxin · 2014-11-11T03:29:51Z

I filed a JIRA for Mima misreporting: https://issues.apache.org/jira/browse/SPARK-4335

cc @pwendell @ScrapCodes

rxin · 2014-11-11T05:27:47Z

Also @ankurdave this needs to be updated now...

Conflicts: graphx/src/main/scala/org/apache/spark/graphx/VertexRDD.scala

SparkQA · 2014-11-11T05:44:51Z

Test build #23197 has started for PR 2530 at commit 1472390.

This patch merges cleanly.

SparkQA · 2014-11-11T07:09:42Z

Test build #23197 has finished for PR 2530 at commit 1472390.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-11T07:09:46Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23197/
Test FAILed.

pwendell · 2014-11-12T07:38:28Z

I took a look and I think MIMA might be reporting a semi-legitimate issue - though this is a tricky situation. I think the issue is - if someone extended the earlier version of EdgeRDD, their class would have inherited the concrete getPartitions call that was present in the earlier version. Now EdgeRDD doesn't have this getPartitions function (it has been moved to the concrete version). I think in general it's fine because this is defined in the parent RDD class, but maybe MIMA doesn't check whether the function is defined further up in the inheritance chain.

In terms of compatiblity for callers, this is a protected method so it really doesn't matter.

I'd go ahead and just add ignores for this and be done with it. Fixing this in MIMA might be diffiuclt

SparkQA · 2014-11-12T07:45:01Z

Test build #23254 has started for PR 2530 at commit d681f45.

This patch merges cleanly.

SparkQA · 2014-11-12T09:05:53Z

Test build #23254 has finished for PR 2530 at commit d681f45.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- abstract class EdgeRDD[ED, VD](
- abstract class VertexRDD[VD](

AmplabJenkins · 2014-11-12T09:05:56Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23254/
Test PASSed.

ankurdave · 2014-11-12T09:27:30Z

@rxin MIMA passed! PTAL.

rxin · 2014-11-12T21:47:05Z

Merging in master & branch-1.2. Thanks!

This discourages users from calling the VertexRDD and EdgeRDD constructor and makes it easier for future changes to ensure backward compatibility. Author: Ankur Dave <[email protected]> Closes #2530 from ankurdave/SPARK-3666 and squashes the following commits: d681f45 [Ankur Dave] Define getPartitions and compute in abstract class for MIMA 1472390 [Ankur Dave] Merge remote-tracking branch 'apache-spark/master' into SPARK-3666 24201d4 [Ankur Dave] Merge remote-tracking branch 'apache-spark/master' into SPARK-3666 cbe15f2 [Ankur Dave] Remove specialized annotation from VertexRDD and EdgeRDD 931b587 [Ankur Dave] Use abstract class instead of trait for binary compatibility 9ba4ec4 [Ankur Dave] Mark (Vertex|Edge)RDDImpl constructors package-private 620e603 [Ankur Dave] Extract VertexRDD interface and move implementation to VertexRDDImpl 55b6398 [Ankur Dave] Extract EdgeRDD interface and move implementation to EdgeRDDImpl (cherry picked from commit a5ef581) Signed-off-by: Reynold Xin <[email protected]>

ankurdave added 2 commits September 24, 2014 19:30

Extract EdgeRDD interface and move implementation to EdgeRDDImpl

55b6398

rxin reviewed Sep 27, 2014
View reviewed changes

Mark (Vertex|Edge)RDDImpl constructors package-private

9ba4ec4

Use abstract class instead of trait for binary compatibility

931b587

Remove specialized annotation from VertexRDD and EdgeRDD

cbe15f2

It makes no difference in benchmarks.

Merge remote-tracking branch 'apache-spark/master' into SPARK-3666

24201d4

Fixes a merge conflict created by ee29ef3. Conflicts: graphx/src/main/scala/org/apache/spark/graphx/EdgeRDD.scala

Merge remote-tracking branch 'apache-spark/master' into SPARK-3666

1472390

Conflicts: graphx/src/main/scala/org/apache/spark/graphx/VertexRDD.scala

ankurdave force-pushed the SPARK-3666 branch from eb9635f to 1472390 Compare November 11, 2014 05:40

Define getPartitions and compute in abstract class for MIMA

d681f45

asfgit closed this in a5ef581 Nov 12, 2014

Earne mentioned this pull request Dec 23, 2014

[SPARK-2823][GraphX]fix GraphX EdgeRDD zipPartitions #1763

Closed

[SPARK-3666] Extract interfaces for EdgeRDD and VertexRDD #2530

[SPARK-3666] Extract interfaces for EdgeRDD and VertexRDD #2530

Conversation

ankurdave commented Sep 25, 2014

SparkQA commented Sep 25, 2014

SparkQA commented Sep 25, 2014

AmplabJenkins commented Sep 25, 2014

rxin Sep 27, 2014

Choose a reason for hiding this comment

rxin commented Sep 27, 2014

SparkQA commented Oct 9, 2014

SparkQA commented Oct 9, 2014

AmplabJenkins commented Oct 9, 2014

ankurdave commented Oct 10, 2014

rxin commented Oct 28, 2014

SparkQA commented Nov 1, 2014

SparkQA commented Nov 1, 2014

SparkQA commented Nov 1, 2014

AmplabJenkins commented Nov 1, 2014

SparkQA commented Nov 2, 2014

SparkQA commented Nov 2, 2014

AmplabJenkins commented Nov 2, 2014

SparkQA commented Nov 2, 2014

AmplabJenkins commented Nov 2, 2014

rxin commented Nov 2, 2014

SparkQA commented Nov 2, 2014

SparkQA commented Nov 2, 2014

AmplabJenkins commented Nov 2, 2014

rxin commented Nov 2, 2014

SparkQA commented Nov 2, 2014

SparkQA commented Nov 2, 2014

AmplabJenkins commented Nov 2, 2014

SparkQA commented Nov 11, 2014

SparkQA commented Nov 11, 2014

rxin commented Nov 11, 2014

rxin commented Nov 11, 2014

SparkQA commented Nov 11, 2014

SparkQA commented Nov 11, 2014

AmplabJenkins commented Nov 11, 2014

pwendell commented Nov 12, 2014

SparkQA commented Nov 12, 2014

SparkQA commented Nov 12, 2014

AmplabJenkins commented Nov 12, 2014

ankurdave commented Nov 12, 2014

rxin commented Nov 12, 2014