[TEP-0094]: Rescope to Task resource UX #588
Conversation
This commit expands the scope of TEP-0094 to cover the user experience of specifying resource requests and limits in Tasks. Focusing only on Step and Sidecar resource requirements may be too narrow of a scope for this TEP. This is largely motivated by tektoncd/pipeline#2986, because the solution to this problem may involve removing the ability to specify Step resource requests. It doesn't make sense to override Step resource requests in TaskRuns if users shouldn't be able to specify Step resource requests in the first place. The scope is also expanded to include parameterizing resource requests based on discussion in tektoncd#560, around treating resource requirement parameterization and runtime overrides as "both/and", rather than "either/or". Fixing Task's resource requirement UX may allow us to get parameterization for free.
tekton-robot
added
the
kind/tep
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
tekton-robot
added
the
size/L
|
/assign |
|
/assign @dibyom |
| ### Confusing Task Resource Requirements UX | ||
| Under the hood, Tekton modifies the resource requests users specify to account for how the Tekton entrypoint runs each `Step` container. | ||
| These modifications aren't transparent to users and have caused confusion. |
There was a problem hiding this comment.
This is kind-of not true 😛. Starting in 0.28 (I think), tekton never modifies resource request users specified, but it will add some resource request/limits on anything that have not been specified by the user, if there is a (or several) LimitRange in the namespace.
| A pod's [effective resource requests and limits](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/#resources) are the higher of: | ||
|
|
||
| - the sum of container and sidecar resource requests/limits | ||
| - the maximum requests/limits of any init container |
There was a problem hiding this comment.
Isn't it the sum of both ? (aka the max of init containers + the sum of container — including sidecars) ?
|
|
||
| This information is used to schedule the pod on a node, and the pod may not be scheduled if its requirements are too large. | ||
| Cluster admins may also define [resource quotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/), which restrict resource consumption | ||
| per namespace, and [limit ranges](https://kubernetes.io/docs/concepts/policy/limit-range/), which restrict resource consumption per pod, container, or PVC. |
There was a problem hiding this comment.
LimitRange are also there to set some default value if the user doesn't set any.
| Tekton `Step`s are run sequentially. This means that the amount of resources needed by the `Task`'s pod is constrained by the maximum `Step` | ||
| resource requests, rather than the sum of the resource requests of all `Step`s. | ||
|
|
||
| Tekton implements this by applying the maximum `Step` resource request to one container, and not applying resource requests to any other container. |
There was a problem hiding this comment.
This is not true anymore though, it doesn't zero out resource request anymore, it uses what the user gave or use LimitRange but even there, never zeros out anything.
| Tekton modifies container resource requirements to comply with any LimitRange policies. For example, if a LimitRange requires a minimum resource | ||
| request, Tekton modifies each container to request at least that minimum. | ||
| See [Presentation: Tekton Resource Requests](https://docs.google.com/presentation/d/1-FNMMbRuxckAInO2aJPtzuINqV-9pr-M25OXQoGbV0s/edit#slide=id.p) | ||
| and [Tekton LimitRange docs](https://github.com/tektoncd/pipeline/blob/main/docs/taskruns.md#specifying-limitrange-values) for more information. |
There was a problem hiding this comment.
It does this on anything the user didn't specify. If a user specify a resource request lower than the allowed value, Tekton would just "let it fail", because it's coming from the user.
| As a result, all `Step` resource requests other than the largest are ignored by Tekton, causing confusion for users. | ||
| In addition, Tekton doesn't currently account for [ResourceQuotas] when determining container resource requests, as documented in | ||
| [Issue: Resource request not applied to init and sidecar containers](https://github.com/tektoncd/pipeline/issues/2933). |
There was a problem hiding this comment.
Same as above, this is not true anymore.
|
|
||
| ### Goals | ||
|
|
||
| - Align `Task` resource requirements API with how `Task`s are scheduled. |
There was a problem hiding this comment.
I am not sure I understand that item, what does it mean ? 🤔
| - Allow parameterization of any resource requirement related fields defined in `Task`, including within `Step`s and `Sidecar`s. | ||
| - Add configuration to `TaskRun` allowing users to modify resource requirements defined in the corresponding `Task`, | ||
| including those defined in `Step`s and `Sidecar`s. |
There was a problem hiding this comment.
I think this is the reason of this "rescope" but those two item can be tackled completely independently of each other (but we can also group them in this TEP).
|
The main reason for rescoping this TEP was that I was concerned users did not like the way of specifying Step resource requirements (described in tektoncd/pipeline#2986), so overriding it wouldn't fix that problem. |
This commit expands the scope of TEP-0094 to cover the user experience of
specifying resource requests and limits in Tasks.
Focusing only on Step and Sidecar resource requirements may be too narrow of a scope for this TEP.
This is largely motivated by tektoncd/pipeline#2986, because the solution
to this problem may involve removing the ability to specify Step resource requests.
It doesn't make sense to override Step resource requests in TaskRuns if users shouldn't be able
to specify Step resource requests in the first place.
The scope is also expanded to include parameterizing resource requests based on discussion in
#560, around treating resource requirement parameterization
and runtime overrides as "both/and", rather than "either/or". Fixing Task's resource requirement UX
may allow us to get parameterization for free.
/kind tep