Task failure during staging leaves orphaned executor, preventing recovery

[dylanwilder]

Executor log shows process has exited, however mesos state shows it as staging. This prevents framework from recovering as it is unable to retrieve the resources.

[mohitsoni]

This is interesting. So, just to clarify: executor never launched ? (because fetcher failed to fetch JRE which is required for executor launch).

[dylanwilder]

I've seen this happen several times as I'm testing a deployment. Most recent shows node-0 and node-0-template in STAGING. Stdout logs show that the child process exited, but the container still hangs around. I've also seen the task in FAILED state yet still under active tasks in Mesos. The framework appears to be trying to recover, but is unable to get the port reservation it needs because there appears to be an orphaned reservation occupying the port range from the failed task. The latter may be an issue with mesos that is just a side effect of the initial error. In any case the only way I'm able to recover is to assassinate the slave machine.

I just wiped an restarted so I could move forward, but if you'd like I can try to get you some logs. I also opened a support ticket with mesosphere before I had narrowed it down to failure to fetch the jre. That ticket contains more details.

[mohitsoni]

@dylanwilder Logs would be helpful. I'll find the ticket. Thanks.

[dylanwilder]

Ok, I've figured out the root cause here (should note i'm on version 1.0.17-3.0.8). There's two issues, the first on the deploy, the second on the inability to recover.

1. I am updating the executor configuration (i.e. to change the jre or cassandra version). This triggers a cluster rolling restart here. Looks like the issue was fixed here, but this used to read getExecutor().withNewId().getExecutorInfo() which caused the updates to be out of sync.
2. This error causes the the task to FAIL, however the executor is orphaned. The scheduler is unaware of it's existence but it is still consuming the executor port resource and the scheduler is unable to relaunch. killing the process loops back to (1)

Seeing as how (1) has been addressed already, is it possible that (2) has been fixed as well?

[dylanwilder]

Looks like this code will always attempt to create a new executor. Really it should be reusing the existing executor, however since we are not actually tracking the state of executors themselves this could lead to reverse problem. It seems to me that executors should really be tracked as first class assets rather than as a reference from tasks. Thoughts?

[nickbp]

Marking closed as #316 was merged