[SPARK-2411] Add a history-not-found page to standalone Master #1336

andrewor14 · 2014-07-08T23:27:33Z

Problem. Right now, if you click on an application after it has finished, it simply refreshes the page if there are no event logs for the application. This is not super intuitive especially because event logging is not enabled by default. We should direct the user to enable this if they attempt to view a SparkUI after the fact without event logs.

Fix. The new page conveys different messages in each of the following scenarios:
(1) Application did not enable event logging,
(2) Event logs are not found in the specified directory, and
(3) Exception is thrown while replaying the logs

Here are screenshots of what the page looks like in each of the above scenarios:

(1)

(2)

(3)

AmplabJenkins · 2014-07-08T23:31:07Z

Merged build triggered.

AmplabJenkins · 2014-07-08T23:31:16Z

Merged build started.

AmplabJenkins · 2014-07-09T00:10:49Z

Merged build finished.

AmplabJenkins · 2014-07-09T00:10:49Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16429/

andrewor14 · 2014-07-09T00:31:59Z

Jenkins test this please

AmplabJenkins · 2014-07-09T00:36:08Z

Merged build triggered.

AmplabJenkins · 2014-07-09T00:36:16Z

Merged build started.

AmplabJenkins · 2014-07-09T00:46:22Z

Merged build finished.

AmplabJenkins · 2014-07-09T00:46:22Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16433/

andrewor14 · 2014-07-09T15:37:40Z

Jenkins test this please

AmplabJenkins · 2014-07-09T15:41:11Z

Merged build triggered.

AmplabJenkins · 2014-07-09T15:41:16Z

Merged build started.

AmplabJenkins · 2014-07-09T17:42:23Z

Merged build finished.

AmplabJenkins · 2014-07-09T17:42:23Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16466/

andrewor14 · 2014-07-09T20:10:37Z

Jenkins test this please

AmplabJenkins · 2014-07-09T20:11:11Z

Merged build triggered.

AmplabJenkins · 2014-07-09T20:11:19Z

Merged build started.

AmplabJenkins · 2014-07-09T21:15:31Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-07-09T21:15:32Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16476/

andrewor14 · 2014-07-10T07:46:31Z

Please don't merge this yet. @aarondav made a comment about the wording offline that I want to address.

andrewor14 · 2014-07-10T21:24:29Z

Provided the tests pass (they should) this is ready from my side.

SparkQA · 2014-07-10T21:27:41Z

QA tests have started for PR 1336. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16524/consoleFull

SparkQA · 2014-07-10T23:12:23Z

QA results for PR 1336:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16524/consoleFull

AmplabJenkins · 2014-07-10T23:12:47Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16524/

pwendell · 2014-07-11T18:55:10Z

core/src/main/scala/org/apache/spark/deploy/master/ui/HistoryNotFoundPage.scala

+    val content =
+      <div class="row-fluid">
+        <div class="span12" style="font-size:14px;font-weight:bold">
+          No event logs were found for this application. To enable event logging, please set


Should you also mention the setting for the event log directory? Otherwise they might think they can just set enabled and it will auto-magically work.

Well, it kind of does auto-magically work, but sure I'll add it

Not sure what you mean, but in most environments users will have to add an HDFS directory there that is shared... right?

Right now the code actually creates the directory for you, so the user doesn't need to add it herself

I mean that if the are joining an existing cluster, they'll need to figure out what HDFS or FS path to set this too such that it's consistent with the path set by the person running the history server.

This is for the standalone Master, not the history server. Here the event log directory for each application is shipped to the master as part of the application description, so the Master knows where to find the logs automatically.

(For history server, however, the user should explicitly set the path in all cases as you describe)

pwendell · 2014-07-11T18:55:21Z

LGTM with one small comment.

andrewor14 · 2014-07-16T21:13:37Z

I have updated the screenshot. Anything else?

pwendell · 2014-07-16T21:19:10Z

Looks good, thanks!

pwendell · 2014-07-16T21:31:28Z

Hey actually - could this differentiate between the case where the event log was not enabled (and give a message like "Application did not enable event logging") and a case where the event log was expected but could not be read (and give a message like "Error reading event log"). Bonus points if you can encode the error message in the second case.

With this commit, the HistoryNotFoundPage behaves differently for each of the following scenarios: (1) User did not enable event logging -> point them to the configs (2) Event logs are not found for some reason -> tell them that (3) Exception is thrown in replaying -> show them the stack trace

SparkQA · 2014-07-17T01:47:57Z

QA tests have started for PR 1336. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16759/consoleFull

SparkQA · 2014-07-17T03:25:17Z

QA results for PR 1336:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16759/consoleFull

andrewor14 · 2014-07-17T17:21:19Z

I have updated the screenshots again. Anything else?

pwendell · 2014-07-17T17:56:30Z

core/src/main/scala/org/apache/spark/deploy/master/Master.scala

-    } else {
-      logWarning("Application %s (%s) has no valid logs: %s".format(appName, app.id, eventLogDir))
+
+    if (eventLogPaths.isEmpty) {


What happens here if eventLogDir doesn't exist?

See L670. If eventLogDir doesn't exist, we never even reach here.

(If you mean the directory does not exist on the filesystem itself, then it does go into this case, and the page displays view (2) in the PR message)

pwendell · 2014-07-18T02:45:43Z

Okay cool - I'm merging tihs.

**Problem.** Right now, if you click on an application after it has finished, it simply refreshes the page if there are no event logs for the application. This is not super intuitive especially because event logging is not enabled by default. We should direct the user to enable this if they attempt to view a SparkUI after the fact without event logs. **Fix.** The new page conveys different messages in each of the following scenarios: (1) Application did not enable event logging, (2) Event logs are not found in the specified directory, and (3) Exception is thrown while replaying the logs Here are screenshots of what the page looks like in each of the above scenarios: (1) <img src="https://issues.apache.org/jira/secure/attachment/12656204/Event%20logging%20not%20enabled.png" width="75%"> (2) <img src="https://issues.apache.org/jira/secure/attachment/12656203/Application%20history%20not%20found.png"> (3) <img src="https://issues.apache.org/jira/secure/attachment/12656202/Application%20history%20load%20error.png" width="95%"> Author: Andrew Or <[email protected]> Closes apache#1336 from andrewor14/master-link and squashes the following commits: 2f06206 [Andrew Or] Merge branch 'master' of github.com:apache/spark into master-link 97cddc0 [Andrew Or] Add different severity levels 832b687 [Andrew Or] Mention spark.eventLog.dir in error message 51980c3 [Andrew Or] Merge branch 'master' of github.com:apache/spark into master-link ded208c [Andrew Or] Merge branch 'master' of github.com:apache/spark into master-link 89d6405 [Andrew Or] Reword message e7df7ed [Andrew Or] Add a history not found page to standalone Master

Add a history not found page to standalone Master

e7df7ed

andrewor14 added 2 commits July 10, 2014 14:23

Reword message

89d6405

Merge branch 'master' of github.com:apache/spark into master-link

ded208c

pwendell reviewed Jul 11, 2014
View reviewed changes

Merge branch 'master' of github.com:apache/spark into master-link

51980c3

Mention spark.eventLog.dir in error message

832b687

andrewor14 added 2 commits July 16, 2014 18:42

Merge branch 'master' of github.com:apache/spark into master-link

2f06206

pwendell reviewed Jul 17, 2014
View reviewed changes

asfgit closed this in 6afca2d Jul 18, 2014

andrewor14 deleted the master-link branch July 21, 2014 20:57

[SPARK-2411] Add a history-not-found page to standalone Master #1336

[SPARK-2411] Add a history-not-found page to standalone Master #1336

Conversation

andrewor14 commented Jul 8, 2014

AmplabJenkins commented Jul 8, 2014

AmplabJenkins commented Jul 8, 2014

AmplabJenkins commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

andrewor14 commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

andrewor14 commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

andrewor14 commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

andrewor14 commented Jul 10, 2014

andrewor14 commented Jul 10, 2014

SparkQA commented Jul 10, 2014

SparkQA commented Jul 10, 2014

AmplabJenkins commented Jul 10, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pwendell commented Jul 11, 2014

andrewor14 commented Jul 16, 2014

pwendell commented Jul 16, 2014

pwendell commented Jul 16, 2014

SparkQA commented Jul 17, 2014

SparkQA commented Jul 17, 2014

andrewor14 commented Jul 17, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pwendell commented Jul 18, 2014