Implement ES daemon-mode in process launcher

[drawlerr]

Launch Elasticsearch as a daemon.

Relates to #697
Closes #718

[danielmitterdorfer]

Thanks for the PR! I left some high-level comments that I think help simplifying the overall approach.

[danielmitterdorfer]

I think testing this gets much simpler once we only rely on the PID. Then we can just mock process startup and mock the check for a PID file. Once we expose a provision and launch subcommand this should also become way simpler to add this as an integration test in integration-test.sh. I'd imagine an integration test along those lines:

PROVISIONER_ID=$(esrally provision --distribution-version=6.8.0 --car=4gheap)
esrally start --provisioner-id=${PROVISIONER_ID}
# maybe do a check here that the node is actually reachable via HTTP
esrally stop --provisioner-id=${PROVISIONER_ID}

[danielmitterdorfer]

Thanks for iterating! I left a few comments but the overall direction looks good to me.

[danielmitterdorfer]

@dliappis can we assume that if docker-compose returns (with exit code 0), Elasticsearch is about to startup? (i.e. no need to wait for the PID file)?

[dliappis]

I am afraid it's a "it depends" answer :)

With docker-compose it's possible to have healthchecks and depending on how they are defined we can safely assume ES is actually listening; this is done for example in

rally/docker/docker-compose-tests.yml

Lines 26 to 30 in 69cb36d

	healthcheck:
	test: curl -f http://localhost:9200
	interval: 5s
	timeout: 2s
	retries: 10

and the other service, which can be a sidecar, depends on it.

[danielmitterdorfer]

@dliappis I think this is entirely in our control as the only way to start the Docker image is via https://github.com/elastic/rally/blob/master/esrally/resources/docker-compose.yml - So we cannot assume it is already listening (which is ok) but I wonder whether we can assume that the Elasticsearch process has been started already?

[danielmitterdorfer]

I left a few comments around _start_process. Also, we usually try to avoid force-pushing during the review process.

[dliappis]

I left some comments on making the Docker Launcher more robust.

[dliappis]

Thanks for iterating! I left some comments, mainly to discuss whether we can rid our selves from the Docker library dependency.

[dliappis]

Thanks for iterating! I spent some time also testing various scenarios and this is looking good.

I left a comment for a bug, I think, that prevents DockerLauncher.stop() from working correctly.

All comments addressed, but Daniel is unavailable to approve.

[dliappis]

This needs to be re-opened (or a new one opened instead) as nightlies fail with:

esrally.exceptions.RallyError: (Daemon startup failed with exit code[1], 'Traceback (most recent call last):\n  File "/var/lib/jenkins/src/rally/esrally/mechanic/mechanic.py", line 576, in receiveMsg_StartNodes\n    nodes = self.mechanic.start_engine()\n  File "/var/lib
/jenkins/src/rally/esrally/mechanic/mechanic.py", line 712, in start_engine\n    self.nodes = self.launcher.start(node_configs)\n  File "/var/lib/jenkins/src/rally/esrally/mechanic/launcher.py", line 282, in start\n    return [self._start_node(node_configuration, node_c
ount_on_host) for node_configuration in node_configurations]\n  File "/var/lib/jenkins/src/rally/esrally/mechanic/launcher.py", line 282, in <listcomp>\n    return [self._start_node(node_configuration, node_count_on_host) for node_configuration in node_configurations]\n
  File "/var/lib/jenkins/src/rally/esrally/mechanic/launcher.py", line 311, in _start_node\n    node_pid = self._start_process(binary_path, env)\n  File "/var/lib/jenkins/src/rally/esrally/mechanic/launcher.py", line 361, in _start_process\n    raise exceptions.LaunchEr
ror(msg)\nesrally.exceptions.LaunchError: (\'Daemon startup failed with exit code[1]\', None)\n')