Windows support to run integration tests with Gradle

[ywelsch]

Implementation note:

The Windows elasticsearch.bat start script does not provide a daemon mode (nor is it easy to add one). As Gradle's Exec task does not support spawning a process in the background, I had to fall back to Ant's exec task. To make things uniform for Windows and Unix, I removed the daemon flag from the Unix shell script invocation. As consequence, the error output is a bit reduced.

Relates to #13930.

[ywelsch]

@rjernst can you have a look?

[rjernst]

In the ant integ tests, we do this logic within the exec code. I think we should do the same (ie in ClusterFormationTasks, where we set the exec args, we can prepend call)?

[ywelsch]

Agree, but I would like to go one step further: The method void plugin(String name, FileCollection file) should just store the <String, FileCollection> pairs. The method void setupCommand(String name, Object... args) can be removed (as it is too broad and OS-dependent). This leaves it up to ClusterFormationTasks.groovy to implement how plugins are actually installed.

[rjernst]

As consequence, the error output is a bit reduced.

This is my main concern. We need to find a way to have the output of the current command (ie starting ES since this is the only one that needs to spawn) output when there is a failure.

There is a longstanding gradle issue for this:
https://issues.gradle.org/browse/GRADLE-1254

And from that, it seems there is a plugin to do exactly what we want?
https://github.com/marcovermeulen/gradle-spawn-plugin

[rjernst]

Nevermind about that plugin, it has some inherent flaws in the api (eg it expects the process to emit something on stdout when it is "ready", and it is meant to just support unix).

Maybe it is worthwhile to use the ant impl as a template? It is not much code and self contained.
https://github.com/apache/ant/blob/master/src/main/org/apache/tools/ant/taskdefs/Exec.java

Even better would be to try and fix this in gradle...the code already has an "isDaemon" flag, but it is completely internal (looks like it is used for the gradle daemon).
https://github.com/gradle/gradle/blob/master/subprojects/core/src/main/groovy/org/gradle/process/internal/ExecHandleRunner.java

[ywelsch]

My understanding of the original implementation is as follows (please correct me if I'm wrong). The daemon flag is added as a parameter to the Gradle Exec task. This means that the Unix shell script spawns the Elasticsearch Java process with exec "$JAVA" ... org.elasticsearch.bootstrap.Elasticsearch start "$@" <&- & and does not wait for the forked process to end. Grade's errorOutput only captures errors in the shell script but not any errors related to the actual Elasticsearch Java process.

The question now is: Why can't we just do the same thing on Windows? First of all, the Windows .bat file does not have a daemon option (yet). This could be implemented, however, using start "Elasticsearch" cmd /c "%JAVA_HOME%\bin\java" .... Seems easy at first but does not work unfortunately. The reason is that child processes spawned using start inherit filehandles stdin, stdout and stderr. This means that Gradle's exec process cannot exit as filehandles of the Elasticsearch Java child process are still open. Unfortunately, there is no easy way to spawn a proper daemon process from a .bat file. The hacky solution for this is to use another bootstrapping executable that starts the actual command using CreateProcess with bInheritHandles set to false. Possible solutions for this are to write a custom executable in C (1), or to use the windows scripting host cscript.exe (2).

1. http://stackoverflow.com/questions/1536205/running-another-program-in-windows-bat-file-and-not-create-child-process
2. https://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+build and
   https://wiki.jenkins-ci.org/download/attachments/1835010/antRunAsync.js

I see two solutions here:

• Support the same error handling behavior for Windows as for Linux. This means implementing a daemon flag in elasticsearch.bat, for example using cscript.exe. Still feels like a hack on Windows.
• Not support the same error handling behavior. The implementation for Linux is kept as is. The Windows batch file is started in ClusterFormationTasks.groovy by using Ant's exec task that supports spawn (Future work would be to add the spawn functionality to Gradle's Exec task). No error output is captured for Windows.

[ywelsch]

The condition was not properly working before with
if (ant.properties.containsKey("failed${task.name}#start")) { because containsKey is called with a GString which does not equal to the represented String.

[rjernst]

Can we do an actual check like .isEmpty() == false? I don't like relying on groovyness here..

[rjernst]

LGTM, thank you for this!

[rjernst]

Maybe add a longer comment in the windows exec block explaining why it is different (and maybe create an issue to follow up on this)?