-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow PVC of completed PipelineRun to be deleted #5776
Comments
If I remember it correctly from implementing |
Hey @jlpettersson - a completed pod no longer has the PVC mounted - easily demonstrated as it doesn't prevent a RWO PVC from being mounted by another node. However the PVC protection feature prevents PVCs from being deleted if referenced by pods, even if in completed state. The linked Argo workflows PR removes the PVC protection finalizer when it knows all the pods are completed and it's safe to do so |
Issues go stale after 90d of inactivity. /lifecycle stale Send feedback to tektoncd/plumbing. |
/remove-lifecycle stale Still an issue |
/remove-lifecycle stale |
Just came across this issue this morning where I didn't expect the PVC to be seen after the I am trying to understand... what is the use of such PVCs? Aren't they unnecessary reserving resources? Shouldn't this be a bug rather than a feature? I think, the /cc @skaegi |
Thanks @skaegi for the references: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#limitations
The statefulset is |
@pritidesai An option to clean up PVCs per PipelineRun would be awesome! I suppose in some cases people might want the option to keep them around for debugging? For various reasons I have to keep completed PipelineRuns around, and if I were using |
Related: #5412 |
PVCs can be specified using volumeClaimTemplates as part of a pipelineRun. The pipelineRun controller creates a new PVC using the specifications from the template. Once the pipelineRun is complete, the PVC exist to faciliate any analysis on the completed pod. Such PVCs are helpful but requires an extra cleanup if not needed. In the cases where pipelineRuns are kept for a long period of time, such PVC can waste resources. This commit introduces an option for the user such that the users can configure pipelineRun controller to delete such PVCs. There is no API change needed to support this cleanup. PipelineRun/TaskRun controller now reads a label from the volumeClaimTemplate to decide whether to delete PVC or not. Set the label " pvc-protection-finalizer" to "remove" to take advantage of this kind of cleanup. kind: PipelineRun metadata: generateName: pipeline-run- spec: workspaces: - name: source volumeClaimTemplate: metadata: labels: pvc-protection-finalizer: remove Closes tektoncd#5776 Signed-off-by: pritidesai <[email protected]>
PVCs can be specified using volumeClaimTemplates as part of a pipelineRun. The pipelineRun controller creates a new PVC using the specifications from the template. Once the pipelineRun is complete, the PVC exist to faciliate any analysis on the completed pod. Such PVCs are helpful but requires an extra cleanup if not needed. In the cases where pipelineRuns are kept for a long period of time, such PVC can waste resources. This commit introduces an option for the user such that the users can configure pipelineRun controller to delete such PVCs. There is no API change needed to support this cleanup. PipelineRun/TaskRun controller now reads a label from the volumeClaimTemplate to decide whether to delete PVC or not. Set the label " pvc-protection-finalizer" to "remove" to take advantage of this kind of cleanup. kind: PipelineRun metadata: generateName: pipeline-run- spec: workspaces: - name: source volumeClaimTemplate: metadata: labels: pvc-protection-finalizer: remove Closes tektoncd#5776 Signed-off-by: pritidesai <[email protected]>
PVCs can be specified using volumeClaimTemplates as part of a pipelineRun. The pipelineRun controller creates a new PVC using the specifications from the template. Once the pipelineRun is complete, the PVC exist to faciliate any analysis on the completed pod. Such PVCs are helpful but requires an extra cleanup if not needed. In the cases where pipelineRuns are kept for a long period of time, such PVC can waste resources. This commit introduces an option for the user such that the users can configure pipelineRun controller to delete such PVCs. There is no API change needed to support this cleanup. PipelineRun/TaskRun controller now reads a label from the volumeClaimTemplate to decide whether to delete PVC or not. Set the label " pvc-protection-finalizer" to "remove" to take advantage of this kind of cleanup. kind: PipelineRun metadata: generateName: pipeline-run- spec: workspaces: - name: source volumeClaimTemplate: metadata: labels: pvc-protection-finalizer: remove Closes tektoncd#5776 Signed-off-by: pritidesai <[email protected]>
Hey @jimmyjones2 I have created a PoC here - #6635 Please let me know if this works for you. Thanks! Once we finalize on the design, I will update the PR to include tests. The changes in the PR are tested locally. |
PVCs can be specified using volumeClaimTemplates as part of a pipelineRun. The pipelineRun controller creates a new PVC using the specifications from the template. Once the pipelineRun is complete, the PVC exist to faciliate any analysis on the completed pod. Such PVCs are helpful but requires an extra cleanup if not needed. In the cases where pipelineRuns are kept for a long period of time, such PVC can waste resources. This commit introduces an option for the user such that the users can configure pipelineRun controller to delete such PVCs. There is no API change needed to support this cleanup. PipelineRun/TaskRun controller now reads a label from the volumeClaimTemplate to decide whether to delete PVC or not. Set the label " pvc-protection-finalizer" to "remove" to take advantage of this kind of cleanup. kind: PipelineRun metadata: generateName: pipeline-run- spec: workspaces: - name: source volumeClaimTemplate: metadata: labels: pvc-protection-finalizer: remove Closes tektoncd#5776 Signed-off-by: pritidesai <[email protected]>
PVCs can be specified using volumeClaimTemplates as part of a pipelineRun. The pipelineRun controller creates a new PVC using the specifications from the template. Once the pipelineRun is complete, the PVC exist to faciliate any analysis on the completed pod. Such PVCs are helpful but requires an extra cleanup if not needed. In the cases where pipelineRuns are kept for a long period of time, such PVC can waste resources. This commit introduces an option for the user such that the users can configure pipelineRun controller to delete such PVCs. There is no API change needed to support this cleanup. PipelineRun/TaskRun controller now reads a label from the volumeClaimTemplate to decide whether to delete PVC or not. Set the label " pvc-protection-finalizer" to "remove" to take advantage of this kind of cleanup. kind: PipelineRun metadata: generateName: pipeline-run- spec: workspaces: - name: source volumeClaimTemplate: metadata: labels: pvc-protection-finalizer: remove Closes tektoncd#5776 Signed-off-by: pritidesai <[email protected]> Signed-off-by: Priti Desai <[email protected]>
Issues go stale after 90d of inactivity. /lifecycle stale Send feedback to tektoncd/plumbing. |
Stale issues rot after 30d of inactivity. /lifecycle rotten Send feedback to tektoncd/plumbing. |
@QuanZhang-William Just tried v0.51.0 with |
Hi @jimmyjones2, #6940 only supports purging PVC with You can see more detail in this discussion |
Sorry @QuanZhang-William I wrote that incorrectly - when I tried |
@jimmyjones2 . Thanks for reporting the bug! With some investigation, the flake is caused by the wrong order of purging finalizer and deleting PVCs. It seems like finalizers cannot be patched/purged when the PVC is not in terminating status, so we need to delete the PVC first and then purge the finalizer. I'm working on a PR to fix it. |
Rotten issues close after 30d of inactivity. /close Send feedback to tektoncd/plumbing. |
@tekton-robot: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/reopen |
@vdemeester: Reopened this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Feature request
For PVCs specified as part of a PipelineRun with
volumeClaimTemplate
:pvc-protection
finalizer from the PVC to allow it to be deleted by the user (similar to Removing PVC finalizer to let it be terminated argoproj/argo-workflows#6629)Use case
If a PVC is specified as part of a PipelineRun with
volumeClaimTemplate
, the PVC lifecycle is the same as the PipelineRun. In the case where PipelineRuns are kept for longer periods this is wasteful of storage resources and some users may want to delete the associated PVC before the PipelineRun, perhaps to keep logs, or because the PipelineRun is managed by a gitops controller.See #1784 for when this was first noticed after PVC protection had been implemented in Kubernetes
The text was updated successfully, but these errors were encountered: