Add a strategy for taints and tolerations

[ravisantoshgudimetla]

Recently one of the users requested a strategy for taints and tolerations. While I don't have cycles to work on this, I would be more than happy to review if anyone in the community is interested to work on.

[paktek123]

any more details, please expand, I would be interested in contributing

[chadswen]

@paktek123 Take a look at this issue for a use case currently implemented with Draino that could be replaced by descheduler: kubernetes/node-problem-detector#199 (comment)

[aveshagarwal]

any more details, please expand, I would be interested in contributing

Lets say a pod was scheduled on a node where there is mismatch between pod's toleration and node's taints, and it could be due to that the pod was scheduled on a node whose taints were later updated. Since, pod's tolerations were only checked at the time admission, and later node's taints were updated, pod continued running on the node, so it only applies for NoSchedule taints.

In summary, it would work as follows:

1. get a list of nods (which already exists)
2. get a list of pods on each node (which already exists)
3. Verify that the node's taints (NoSchedule) are still satisfied by its pods' tolerations.
4. As per previous step, If not satisfied, evict the pod.
5. If yes (still satisfied), continue checking other pods until you have reached the end of list of pods for that node.
6. Repeat above for all nodes.

[paktek123]

Thanks for the explanation, I will hopefully try to contribute next week

[warmchang]

Nice idea! It is also very useful for customizing the migration time of POD.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[damemi]

/reopen

[k8s-ci-robot]

@damemi: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[isindir]

/reopen

[k8s-ci-robot]

@isindir: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[seanmalloy]

/reopen

[k8s-ci-robot]

@seanmalloy: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[seanmalloy]

/remove-lifecycle rotten
/kind feature

[pravarag]

@seanmalloy @damemi looks like this issue hasn't been picked up in long time. Is it okay if I give it a try? will post my queries here if any.

/assign

[StevenACoffman]

Please do!

[pravarag]

@damemi @aveshagarwal one query around this description mentioned here, Are we looking to add a new check in api itself like somewhere here: https://github.com/kubernetes-sigs/descheduler/tree/master/pkg/descheduler/node or this feature is going to be a new change w.r.t command line options as well maybe here: https://github.com/kubernetes-sigs/descheduler/tree/master/cmd/descheduler/app ?

[damemi]

I'm not actually sure why this issue is still open, I might have mistakenly reopened it when it went stale... we have a Taints/Tolerations strategy that was merged out of this issue (https://github.com/kubernetes-sigs/descheduler#removepodsviolatingnodetaints)

Is there something that strategy is missing from the discussion here? Or can we close this?

[StevenACoffman]

Just checking, but is it now possible to:

1. Detect permanent node problems and set Node Conditions using the Node Problem Detector and the scheduler's TaintNodesByCondition functionality.
2. Configure Descheduler to deschedule pods based on taints to cordon and drain nodes when they exhibit the NPD's KernelDeadlock condition, or a variant of KernelDeadlock we call VolumeTaskHung.
3. Let the Cluster Autoscaler scale down underutilised nodes, including the nodes Descheduler has drained.

If so, is there an example?

[damemi]

@StevenACoffman yes, the descheduler will evict any pods that are currently running on a node that has any NoSchedule taint that the pods do not tolerate. So, your use case should work with the RemovePodsViolatingNodeTaints strategy

[pravarag]

Thanks @damemi for clarifying, I guess this issue will be closed now 🙂

[damemi]

/close

[k8s-ci-robot]

@damemi: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.