Support HTTP operation retry with exponential backoff (for status code >= 500)

[attilapiros]

According to the API conventions when the HTTP status code is one of the followings:

• 500 StatusInternalServerError
• 503 StatusServiceUnavailable
• 504 StatusServerTimeout
  Then the suggested client recovery behavior is "retry with exponential backoff".

(In case of 504 it is "Increase the value of the timeout param and retry with exponential backoff". But in the first PR I would focus on just on the retries).

These is valid problem I faced in a production system in Apache Spark but looked around and I found several some other cases where the root cause was the same.

In Apache Spark it was caused by etcd server leader election:

Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: 
https://kubernetes.default.svc/api/v1/namespaces/dex-app-bl24w4z9/pods/sparkpi-10-fcd3f6781a874212-driver. Message: 
etcdserver: leader changed. Received status: Status(apiVersion=v1, code=500, details=null, kind=Status, message=etcdserver: 
leader changed, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, 
additionalProperties={}), reason=null, status=Failure, additionalProperties={}).

But other applications are suffering from similar issues this is why I think we can provide a solution here in the Kubernetes Client.

My solution would introduce 2 new configs:

• kubernetes.request.retry.backoffLimit
• kubernetes.request.retry.backoffInterval

Where "kubernetes.request.retry.backoffLimit" by default would be 0 to keep backward compatibility.

[attilapiros]

I am working on the PR. But has no right to assign the issue to myself.

[attilapiros]

Thanks @rohanKanojia!

[rohanKanojia]

Which operations would be affected by this change?

[attilapiros]

I would do this in OperationSupport#handleResponse() so all of them.

[attilapiros]

Regarding that method it would be this elegant:

-    Response response = client.newCall(request).execute();
+    Response response = retryWithExponentialBackoff(client, request);