Is there or will there ever be metric for pods in terminating state?

[cullepl]

I see there are metrics for if a pod is terminated kube_pod_container_status_terminated however, we sometimes observe pods in terminating state.

I was wondering if there is a way via kube-state-metrics to find this state? I couldnt see anything and I'm running v1.2.0

All I see in our prometheus are these:

If pods are stuck in terminating state then we usually have to take actions against the hosts they are running on.
We'd ideally like a solution via prometheus and alertmanager using kube-state-metrics metrics to alert us rather than having to create something home grown.

[brancz]

How are you determining this today?

If kube-state-metrics can find what you are looking for in a Pod object then yes, that's something we could add a metric for.

[cullepl]

If I query pods using kubectl get pods -n my-namespace today, I see this:

my-namespace  my-pod-1   6/6       Terminating   1          11d       <IP>   1.2.3.4
my-namespace   my-pod-2   1/6       Terminating   22         10d       <none>           1.2.3.5

If I describe one of the PODs, the status is

Status:				Terminating (expires Sat, 13 Jan 2018 16:35:28 +0000)

[brancz]

When you view the yaml of this pod, can you see it there as well, or is this only available through events?

[cullepl]

Looks like it's only through describing the pod.
If i do kubectl get pod -n <ns> <pod-name> -o yaml there is no reference to terminating. At the end of the yaml output there is

  phase: Running
  podIP: 1.2.3.4
  startTime: 2018-01-06T06:21:34Z

[brancz]

Yes that's what I thought. The tricky thing here is that we can only extract this information from Kubernetes Events. The problem with events is their cardinality, it is most likely going to explode. Something we could do is aggregate events and expose lower cardinality aggregations. This would definitely be a very different pattern from all the other collectors today, where the objects we create metrics from are in memory. With events we would need to listen to events and garbage collect them if they're older than some period.

We could try this as an experiment, but my general feeling is that this is going to be somewhat unstable data.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[naveenb29]

/reopen /remove-lifecycle rotten

[nucl3arj4zz]

Will this issue be re-opened? /reopen

[mxinden]

Given the fact that this information is not extractable from a pod manifest (see comment), this can not be achieved with the current architecture of kube-state-metrics.

I am not sure we should introduce event aggregation here, just to support this use case. What are your thoughts?