Simple HTTP based healthcheck implementation for cloud environments that support rolling restarts like Openshift or Google Cloud Engine

[noctarius]

This PR adds a very basic health check to the Hazelcast HTTP subsystem that provides information about the current health status (ongoing migrations, cluster size, ...). That way monitoring solutions or ready / health checks can be implemented on the cloud environment side.

The information provided are minimal to not interfere too much with the cluster load. Requests to http://xxx.xxx.xxx.xxx:5701/hazelcast/health are accepted whenever the property hazelcast.http.healthcheck.enabled is set to true. In this case a GET request provides a result like the following:

Hazelcast::NodeState=ACTIVE
Hazelcast::ClusterState=ACTIVE
Hazelcast::ClusterSafe=TRUE
Hazelcast::MigrationQueueSize=0
Hazelcast::ClusterSize=1

Based on this result the check can make assumptions like "the cluster is not yet safe, I cannot go on with rolling restarts".

[devOpsHazelcast]

Test FAILed.
https://hazelcast-l337.ci.cloudbees.com/job/new-lab-fast-pr/5793/

[noctarius]

run-lab-run

[devOpsHazelcast]

Test PASSed.

[pveentjer]

Looks good. Can you add some tests?

[noctarius]

How can I test that? I mean it's disabled in mocks, so I would have to create a real cluster node which is bad for parallel testing. Any idea?

[pveentjer]

You can also make it a nightly test and have it run in isolation.

[devOpsHazelcast]

Test PASSed.

[noctarius]

Can we get this merged? :)

[pveentjer]

👍

[devOpsHazelcast]

Test FAILed.
https://hazelcast-l337.ci.cloudbees.com/job/new-lab-fast-pr/5972/

[noctarius]

run-lab-run

[devOpsHazelcast]

Test PASSed.

[mesutcelik]

Thanks @noctarius !

Can we update documentation with that API addition? cc: @Serdaro