[Fault tolerance] Make executor actor restartable and fetch data from blockManager in actor calls #249
Conversation
|
If we use a normal task, which calls the actor internally, the actor method still need to be resubmitted because it's a dependency. I'm not sure if using spark to call the executors will work, but I think sparkcontext would be needed to do so. |
Signed-off-by: Zhi Lin <[email protected]>
Signed-off-by: Zhi Lin <[email protected]>
Signed-off-by: Zhi Lin <[email protected]>
make RayAppMaster an actor enable cache locations to have better data locality Signed-off-by: Zhi Lin <[email protected]>
Signed-off-by: Zhi Lin <[email protected]>
kira-lin
changed the title
Signed-off-by: Zhi Lin <[email protected]>
Signed-off-by: Zhi Lin <[email protected]>
Signed-off-by: Zhi Lin <[email protected]>
Signed-off-by: Zhi Lin <[email protected]>
Signed-off-by: Zhi Lin <[email protected]>
Signed-off-by: Zhi Lin <[email protected]>
Signed-off-by: Zhi Lin <[email protected]>
Signed-off-by: Zhi Lin <[email protected]>
| @@ -138,6 +142,12 @@ private[spark] class ApplicationInfo( | |||
| registeredExecutors | |||
| } | |||
|
|
|||
| def isRemovedExecutor(executorId: String): Boolean = { | |||
There was a problem hiding this comment.
Is this used anywhere?
There was a problem hiding this comment.
This was used in previous implementation. I'll remove it
| val memory = appInfo.desc.memoryPerExecutorMB | ||
| val newExecutorId = s"${appInfo.getNextExecutorId()}" | ||
| // ray actor will restart using the old ID | ||
| val handlerOpt = Ray.getActor("raydp-executor-" + executorId) |
There was a problem hiding this comment.
What if the executor failed twice and restart? In the second time, the executor Id is not the original Id and there is a problem to get the actor? Can you add a test for this case?
There was a problem hiding this comment.
No matter how many time it has failed, it is restarted using the parameters given when first created. The name won't change, either.
There was a problem hiding this comment.
After the first restart, I saw the executorId was modified in RayCoarseGrainedExecutorBackend? Where is the original Id stored and can be got?
There was a problem hiding this comment.
Yes, it is changed. The original id is not stored in executor. It is saved in a mapping in RayAppMaster restartedExecutors. Ray also stores it in task's lineage.
| if (appInfo.remainingUnRegisteredExecutors > 0) { | ||
| val cores = appInfo.desc.coresPerExecutor.getOrElse(1) | ||
| val memory = appInfo.desc.memoryPerExecutorMB | ||
| val newExecutorId = s"${appInfo.getNextExecutorId()}" |
There was a problem hiding this comment.
Should we move this line into else {}? Otherwise if we go to the if {} code path, we don't need this new executor Id but we still increase the Id.
| def getRDDPartition(rddId: Int, | ||
| partitionId: Int, | ||
| schemaStr: String, | ||
| driverAgentUrl: String): Array[Byte] = { |
There was a problem hiding this comment.
can we follow Spark's code style for the indent? Please also check other places in the code. see https://github.com/databricks/scala-style-guide#indent
| from py4j.java_gateway import JavaGateway, GatewayParameters | ||
| from py4j.java_gateway import java_import, JavaGateway, GatewayParameters | ||
| from py4j.clientserver import ClientServer, JavaParameters, PythonParameters | ||
| from pyspark.find_spark_home import _find_spark_home |
There was a problem hiding this comment.
It seems this is not used. Can you please remove it?
Signed-off-by: Zhi Lin <[email protected]>
Signed-off-by: Zhi Lin <[email protected]>
Signed-off-by: Zhi Lin <[email protected]>
Signed-off-by: Zhi Lin <[email protected]>
Signed-off-by: Zhi Lin <[email protected]>
Signed-off-by: Zhi Lin <[email protected]>
Signed-off-by: Zhi Lin <[email protected]>
Signed-off-by: Zhi Lin <[email protected]>
Signed-off-by: Zhi Lin <[email protected]>
Signed-off-by: Zhi Lin <[email protected]>
Signed-off-by: Zhi Lin <[email protected]>
Signed-off-by: Zhi Lin <[email protected]>
Signed-off-by: Zhi Lin <[email protected]>
Signed-off-by: Zhi Lin <[email protected]>
Signed-off-by: Zhi Lin <[email protected]>
No description provided.