Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SPARK-1104: kill Process in workerThread of ExecutorRunner #35

Closed
wants to merge 5 commits into from
Closed

SPARK-1104: kill Process in workerThread of ExecutorRunner #35

Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
1d511c8
kill Process in workerThread
CodingCat Feb 27, 2014
0accf2f
set process to null after killed it
CodingCat Feb 27, 2014
eb615ba
kill the process when the error happens
CodingCat Mar 4, 2014
3107aeb
address Aaron's comments
CodingCat Apr 24, 2014
85767da
add null checking and remove unnecessary killProce
CodingCat Apr 24, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 13 additions & 17 deletions core/src/main/scala/org/apache/spark/deploy/worker/ExecutorRunner.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,30 +58,27 @@ private[spark] class ExecutorRunner(
override def run() { fetchAndRunExecutor() }
}
workerThread.start()

// Shutdown hook that kills actors on shutdown.
shutdownHook = new Thread() {
override def run() {
if (process != null) {
logInfo("Shutdown hook killing child process.")
process.destroy()
process.waitFor()
}
killProcess()
}
}
Runtime.getRuntime.addShutdownHook(shutdownHook)
}

private def killProcess() {
logInfo("Killing process!")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we still should have the null check (just not null set) since process starts out null, like in the prior shutdown hook.

process.destroy()
process.waitFor()
}

/** Stop this executor runner, including killing the process it launched */
def kill() {
if (workerThread != null) {
// the workerThread will kill the child process when interrupted
workerThread.interrupt()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a comment that the worker thread will the child process when interrupted.

workerThread = null
if (process != null) {
logInfo("Killing process!")
process.destroy()
process.waitFor()
}
state = ExecutorState.KILLED
worker ! ExecutorStateChanged(appId, execId, state, None, None)
Runtime.getRuntime.removeShutdownHook(shutdownHook)
Expand Down Expand Up @@ -126,7 +123,6 @@ private[spark] class ExecutorRunner(
// parent process for the executor command
env.put("SPARK_LAUNCH_WITH_SCALA", "0")
process = builder.start()

val header = "Spark Executor Command: %s\n%s\n\n".format(
command.mkString("\"", "\" \"", "\""), "=" * 40)

Expand All @@ -142,18 +138,18 @@ private[spark] class ExecutorRunner(
// long-lived processes only. However, in the future, we might restart the executor a few
// times on the same machine.
val exitCode = process.waitFor()
killProcess()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The contract of waitFor says the process should be terminated by the end -- is there a need to killProcess() it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aarondav en , I think we don't need it now, I originally just wanted to set process to null.....

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can it be removed now that we don't set process to null there?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

....I think I forgot to do that after I commented......

state = ExecutorState.FAILED
val message = "Command exited with code " + exitCode
worker ! ExecutorStateChanged(appId, execId, state, Some(message), Some(exitCode))
} catch {
case interrupted: InterruptedException =>
case interrupted: InterruptedException => {
logInfo("Runner thread for executor " + fullId + " interrupted")

killProcess()
}
case e: Exception => {
logError("Error running executor", e)
if (process != null) {
process.destroy()
}
killProcess()
state = ExecutorState.FAILED
val message = e.getClass + ": " + e.getMessage
worker ! ExecutorStateChanged(appId, execId, state, Some(message), None)
Expand Down