consul lock may spawn the watched process at shutdown

[mfischer-zd]

We saw this unusual chain of events reported by consul lock after sending it a SIGTERM today:

2015-08-03_22:39:06.20297 consul:Setting up lock at path: xxx
2015-08-03_22:39:06.20308 consul:Attempting lock acquisition
# Lock was never acquired because another candidate held it

# SIGTERM was then sent
2015-08-04_19:56:20.23976 consul:Shutdown triggered, killing child
2015-08-04_19:56:20.23988 consul:No child process to kill
# Correct, but then...
2015-08-04_19:56:20.23992 consul:Starting handler 'myapp'
# Wait, what?
2015-08-04_19:56:20.25634 consul:Cleanup succeeded

After this, consul exited and myapp was orphaned and reparented to PID 1.

[mfischer-zd]

I've investigated this matter further and it appears to be a race condition that can occur when a SIGTERM is sent to multiple consul lock processes at approximately the same time.

The short story is this:

1. host1 (current lock holder) receives SIGTERM and releases the lock
2. host2 acquires the lock and runs LockCommand.startChild() in a goroutine
3. host2: in startChild(), cmd.Start() is reached, forks and execs app
4. host2 receives SIGTERM
5. host2 calls LockCommand.killChild()
6. host2: in killChild() tries to evaluate cmd.Process, but it hasn't been set yet in the goroutine running startChild(), so app isn't sent the signal (startChild() returns early)
7. host2 exits, leaving an orphaned app