[SPARK-7501] [Streaming] DAG visualization: show DStream operations #6034

andrewor14 · 2015-05-09T19:22:06Z

This is similar to #5999, but for streaming. Roughly 200 lines are tests.

One thing to note here is that we already do some kind of scoping thing for call sites, so this patch adds the new RDD operation scoping logic in the same place. Also, this patch adds a try finally block to set the relevant variables in a safer way.

@tdas @zsxwing

Before

After

AmplabJenkins · 2015-05-09T19:27:11Z

Merged build triggered.

AmplabJenkins · 2015-05-09T19:27:20Z

Merged build started.

SparkQA · 2015-05-09T19:29:03Z

Test build #32313 has started for PR 6034 at commit a0d3263.

AmplabJenkins · 2015-05-09T21:52:11Z

Merged build triggered.

AmplabJenkins · 2015-05-09T21:52:16Z

Merged build started.

SparkQA · 2015-05-09T21:53:53Z

Test build #32316 has started for PR 6034 at commit e85b787.

SparkQA · 2015-05-09T21:59:04Z

Test build #32313 timed out for PR 6034 at commit a0d3263 after a configured wait of 150m.

AmplabJenkins · 2015-05-09T21:59:09Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-09T21:59:09Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32313/
Test FAILed.

SparkQA · 2015-05-09T23:34:56Z

Test build #32316 has finished for PR 6034 at commit e85b787.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-09T23:35:01Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-09T23:35:02Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32316/
Test PASSed.

tdas · 2015-05-12T00:21:48Z

core/src/main/scala/org/apache/spark/rdd/RDD.scala

+          checkpointData.get.doCheckpoint()
+        } else {
+          dependencies.foreach(_.rdd.doCheckpoint())
+        }


This is not for this PR. The other PR still not merged in master .. isnt it?

oops I can't believe I accidentally merged these two PRs.

tdas · 2015-05-12T00:24:41Z

I think there should be a few unit tests to test this. See how the creation sites of the DStream are tested in StreamingContextSuite, test "DStream and generated RDD creation sites"

tdas · 2015-05-12T00:28:23Z

streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala

+   * Instead, every time we call `compute` we instantiate a new scope using the same name as this
+   * one. Otherwise, all RDDs ever created by this DStream will be in the same scope.
+   */
+  private val scope: Option[RDDOperationScope] = {


Can you store only the name in as a DStream field? I does not make sense to save this arbitrary mostly unnecessary RDDOperationScope object to be serialized with DStreams as part of DStream checkpoints.

tdas · 2015-05-12T00:34:38Z

This doesnt yet set the scope for input stream (that is DStream that are directly created, not using DStream transformations). I think you need to use the withScope in places like StreamingContext.socketStream(). Example:

class StreamingContext {

def socketTextStream(): DStream[String] = withScope {
   new SocketInputDStream[String](...)
}

This needs to be done for every subclass of InputDStream and ReceiverInputDStream

AmplabJenkins · 2015-05-13T21:12:11Z

Merged build triggered.

AmplabJenkins · 2015-05-13T21:12:19Z

Merged build started.

SparkQA · 2015-05-13T21:14:07Z

Test build #32650 has started for PR 6034 at commit c121047.

This covers all the places where we instantiate DStreams.

SparkQA · 2015-05-13T23:03:31Z

Test build #32650 has finished for PR 6034 at commit c121047.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DoubleArrayParam(parent: Params, name: String, doc: String, isValid: Array[Double] => Boolean)
- class Bucketizer(JavaTransformer, HasInputCol, HasOutputCol):

AmplabJenkins · 2015-05-13T23:03:37Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-13T23:03:37Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32650/
Test PASSed.

AmplabJenkins · 2015-05-17T08:42:10Z

Merged build triggered.

AmplabJenkins · 2015-05-17T08:42:16Z

Merged build started.

SparkQA · 2015-05-17T08:44:05Z

Test build #32933 has started for PR 6034 at commit 84d0656.

SparkQA · 2015-05-17T10:28:30Z

Test build #32933 has finished for PR 6034 at commit 84d0656.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-17T10:28:35Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-17T10:28:35Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32933/
Test PASSed.

AmplabJenkins · 2015-05-18T18:07:12Z

Merged build triggered.

AmplabJenkins · 2015-05-18T18:07:20Z

Merged build started.

SparkQA · 2015-05-18T18:09:27Z

Test build #33002 has started for PR 6034 at commit 932a64a.

tdas · 2015-05-18T18:10:12Z

LGTM! Thanks @andrewor14 for this patch.

SparkQA · 2015-05-18T19:20:00Z

Test build #33002 has finished for PR 6034 at commit 932a64a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-18T19:20:05Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-18T19:20:05Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33002/
Test FAILed.

andrewor14 · 2015-05-18T19:35:57Z

retest this please

AmplabJenkins · 2015-05-18T19:37:12Z

Merged build triggered.

AmplabJenkins · 2015-05-18T19:37:21Z

Merged build started.

SparkQA · 2015-05-18T19:39:26Z

Test build #33009 has started for PR 6034 at commit 932a64a.

SparkQA · 2015-05-18T21:31:29Z

Test build #33009 has finished for PR 6034 at commit 932a64a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-18T21:31:35Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-18T21:31:35Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33009/
Test PASSed.

andrewor14 · 2015-05-18T21:33:11Z

Great, merging into master 1.4.

This is similar to #5999, but for streaming. Roughly 200 lines are tests. One thing to note here is that we already do some kind of scoping thing for call sites, so this patch adds the new RDD operation scoping logic in the same place. Also, this patch adds a `try finally` block to set the relevant variables in a safer way. tdas zsxwing ------------------------ **Before** <img src="https://cloud.githubusercontent.com/assets/2133137/7625996/d88211b8-f9b4-11e4-90b9-e11baa52d6d7.png" width="450px"/> -------------------------- **After** <img src="https://cloud.githubusercontent.com/assets/2133137/7625997/e0878f8c-f9b4-11e4-8df3-7dd611b13c87.png" width="650px"/> Author: Andrew Or <[email protected]> Closes #6034 from andrewor14/dag-viz-streaming and squashes the following commits: 932a64a [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-streaming e685df9 [Andrew Or] Rename createRDDWith 84d0656 [Andrew Or] Review feedback 697c086 [Andrew Or] Fix tests 53b9936 [Andrew Or] Set scopes for foreachRDD properly 1881802 [Andrew Or] Refactor DStream scope names again af4ba8d [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-streaming fd07d22 [Andrew Or] Make MQTT lower case f6de871 [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-streaming 0ca1801 [Andrew Or] Remove a few unnecessary withScopes on aliases fa4e5fb [Andrew Or] Pass in input stream name rather than defining it from within 1af0b0e [Andrew Or] Fix style 074c00b [Andrew Or] Review comments d25a324 [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-streaming e4a93ac [Andrew Or] Fix tests? 25416dc [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-streaming 9113183 [Andrew Or] Add tests for DStream scopes b3806ab [Andrew Or] Fix test bb80bbb [Andrew Or] Fix MIMA? 5c30360 [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-streaming 5703939 [Andrew Or] Rename operations that create InputDStreams 7c4513d [Andrew Or] Group RDDs by DStream operations and batches bf0ab6e [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-streaming 05c2676 [Andrew Or] Wrap many more methods in withScope c121047 [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-streaming 65ef3e9 [Andrew Or] Fix NPE a0d3263 [Andrew Or] Scope streaming operations instead of RDD operations (cherry picked from commit b93c97d) Signed-off-by: Andrew Or <[email protected]>

This is similar to apache#5999, but for streaming. Roughly 200 lines are tests. One thing to note here is that we already do some kind of scoping thing for call sites, so this patch adds the new RDD operation scoping logic in the same place. Also, this patch adds a `try finally` block to set the relevant variables in a safer way. tdas zsxwing ------------------------ **Before** <img src="https://cloud.githubusercontent.com/assets/2133137/7625996/d88211b8-f9b4-11e4-90b9-e11baa52d6d7.png" width="450px"/> -------------------------- **After** <img src="https://cloud.githubusercontent.com/assets/2133137/7625997/e0878f8c-f9b4-11e4-8df3-7dd611b13c87.png" width="650px"/> Author: Andrew Or <[email protected]> Closes apache#6034 from andrewor14/dag-viz-streaming and squashes the following commits: 932a64a [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-streaming e685df9 [Andrew Or] Rename createRDDWith 84d0656 [Andrew Or] Review feedback 697c086 [Andrew Or] Fix tests 53b9936 [Andrew Or] Set scopes for foreachRDD properly 1881802 [Andrew Or] Refactor DStream scope names again af4ba8d [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-streaming fd07d22 [Andrew Or] Make MQTT lower case f6de871 [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-streaming 0ca1801 [Andrew Or] Remove a few unnecessary withScopes on aliases fa4e5fb [Andrew Or] Pass in input stream name rather than defining it from within 1af0b0e [Andrew Or] Fix style 074c00b [Andrew Or] Review comments d25a324 [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-streaming e4a93ac [Andrew Or] Fix tests? 25416dc [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-streaming 9113183 [Andrew Or] Add tests for DStream scopes b3806ab [Andrew Or] Fix test bb80bbb [Andrew Or] Fix MIMA? 5c30360 [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-streaming 5703939 [Andrew Or] Rename operations that create InputDStreams 7c4513d [Andrew Or] Group RDDs by DStream operations and batches bf0ab6e [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-streaming 05c2676 [Andrew Or] Wrap many more methods in withScope c121047 [Andrew Or] Merge branch 'master' of github.com:apache/spark into dag-viz-streaming 65ef3e9 [Andrew Or] Fix NPE a0d3263 [Andrew Or] Scope streaming operations instead of RDD operations

Scope streaming operations instead of RDD operations

a0d3263

tdas reviewed May 12, 2015
View reviewed changes

Andrew Or added 2 commits May 13, 2015 14:08

Fix NPE

65ef3e9

Merge branch 'master' of github.com:apache/spark into dag-viz-streaming

c121047

andrewor14 force-pushed the dag-viz-streaming branch from e85b787 to c121047 Compare May 13, 2015 21:09

Wrap many more methods in withScope

05c2676

This covers all the places where we instantiate DStreams.

Merge branch 'master' of github.com:apache/spark into dag-viz-streaming

bf0ab6e

Review feedback

84d0656

Andrew Or added 2 commits May 18, 2015 11:02

Rename createRDDWith

e685df9

Merge branch 'master' of github.com:apache/spark into dag-viz-streaming

932a64a

asfgit closed this in b93c97d May 18, 2015

andrewor14 deleted the dag-viz-streaming branch May 18, 2015 21:46

[SPARK-7501] [Streaming] DAG visualization: show DStream operations #6034

[SPARK-7501] [Streaming] DAG visualization: show DStream operations #6034

Conversation

andrewor14 commented May 9, 2015

AmplabJenkins commented May 9, 2015

AmplabJenkins commented May 9, 2015

SparkQA commented May 9, 2015

AmplabJenkins commented May 9, 2015

AmplabJenkins commented May 9, 2015

SparkQA commented May 9, 2015

SparkQA commented May 9, 2015

AmplabJenkins commented May 9, 2015

AmplabJenkins commented May 9, 2015

SparkQA commented May 9, 2015

AmplabJenkins commented May 9, 2015

AmplabJenkins commented May 9, 2015

tdas May 12, 2015

Choose a reason for hiding this comment

andrewor14 May 12, 2015

Choose a reason for hiding this comment

tdas commented May 12, 2015

tdas May 12, 2015

Choose a reason for hiding this comment

tdas commented May 12, 2015

AmplabJenkins commented May 13, 2015

AmplabJenkins commented May 13, 2015

SparkQA commented May 13, 2015

SparkQA commented May 13, 2015

AmplabJenkins commented May 13, 2015

AmplabJenkins commented May 13, 2015

AmplabJenkins commented May 17, 2015

AmplabJenkins commented May 17, 2015

SparkQA commented May 17, 2015

SparkQA commented May 17, 2015

AmplabJenkins commented May 17, 2015

AmplabJenkins commented May 17, 2015

AmplabJenkins commented May 18, 2015

AmplabJenkins commented May 18, 2015

SparkQA commented May 18, 2015

tdas commented May 18, 2015

SparkQA commented May 18, 2015

AmplabJenkins commented May 18, 2015

AmplabJenkins commented May 18, 2015

andrewor14 commented May 18, 2015

AmplabJenkins commented May 18, 2015

AmplabJenkins commented May 18, 2015

SparkQA commented May 18, 2015

SparkQA commented May 18, 2015

AmplabJenkins commented May 18, 2015

AmplabJenkins commented May 18, 2015

andrewor14 commented May 18, 2015