Add livenessProbe support for kube-rbac-proxy

[jessehu]

When using kube-rbac-proxy v0.14.1, sometimes we found kube-rbac-proxy container stuck in TLS handshake error and can not recover automatically until it's restarted manually.

I0621 17:39:33.819787    1876 log.go:198] http: TLS handshake error from 10.255.9.20:55542: write tcp 10.255.9.20:9100->10.255.9.20:55542: write: broken pipe
I0621 17:40:40.388479    1876 log.go:198] http: TLS handshake error from 10.255.9.26:1531: write tcp 10.255.9.20:9100->10.255.9.26:1531: write: broken pipe
I0621 17:44:32.288256    1876 log.go:198] http: TLS handshake error from 10.255.9.26:30302: write tcp 10.255.9.20:9100->10.255.9.26:30302: write: broken pipe

When used with prometheus node-exporter, adding a livenessProbe can automatically restart kube-rbac-proxy container, e.g.:

        - image: myregistry.io/kube-rbac-proxy:v0.14.1-with-curl
          livenessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - "curl -sSL -ik -H \"Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\" https://${IP}:9100/metrics | grep -e 'HTTP/2 200'"
            initialDelaySeconds: 30
            failureThreshold: 3
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 3

[peterbueschel]

Hi,

I would also vote for either such an option to probe the proxy (but that implies an extra binary in the image) or to maybe panic on such an error.

In our case we also receive:

webhook.go:154] Failed to make webhook authenticator request: Post ".....:443/apis/authentication.k8s.io/v1/tokenreviews": context deadline exceeded 

Afterwards only a manual restart helps to fix that...

[Waji-97]

I have got the same issue using kube-rbac-proxy container with node-exporter in a single pod. To be honest, I am unsure why it occurs but whenever it does, it uses almost all of the CPU & Mem resources limited to the kube-rbac-proxy container and goes back to normal once the container is restarted. Is there any new news on this particular issue?