Add ability to delete a statefulset with the cascade orphan option #1831
Assignees
Labels
area/resource-management
Issues related to Kubernetes resource provisioning, management, await logic, and semantics generally
kind/enhancement
Improvements or new features
resolution/fixed
This issue was fixed
Comments
leezen
added
the
area/resource-management
EronWright
added a commit
that referenced
this issue
May 17, 2024
<!--Thanks for your contribution. See [CONTRIBUTING](CONTRIBUTING.md)
for Pulumi's contribution guidelines.
Help us merge your changes more quickly by adding more details such
as labels, milestones, and reviewers.-->
### Proposed changes
<!--Give us a brief description of what you've done and what it solves.
-->
Introduces a new Pulumi annotation to set the [deletion propagation
policy](https://kubernetes.io/docs/tasks/administer-cluster/use-cascading-deletion/),
e.g. to support non-cascading delete on `StatefulSet` to preserve pods
during replacement (see
[walkthrough](https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#non-cascading-delete)).
Note that the policy annotation must be set on the old resource before
deleting or replacing it; setting it on the replacement or on the live
object is ineffective.
### Related issues (optional)
<!--Refer to related PRs or issues: #1234, or 'Fixes #1234' or 'Closes
#1234'.
Or link to full URLs to issues or pull requests in other GitHub
repositories. -->
Closes #1831
### Example
This example serves to show how the 'orphan' propagation policy allows
for non-disruptive replacement of
a StatefulSet, e.g. by touching the `volumeClaimTemplates`.
```yaml
name: issue-1831
runtime: yaml
description: A StatefulSet to demonstrate the `pulumi.com/deletionPropagationPolicy` annotation.
config:
# disable SSA for this demonstration
kubernetes:enableServerSideApply: false
resources:
nginx:
type: kubernetes:apps/v1:StatefulSet
properties:
metadata:
name: nginx
annotations:
pulumi.com/deletionPropagationPolicy: "orphan"
spec:
replicas: 1
selector:
matchLabels:
app: nginx
serviceName: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- image: nginx:1.19.9
name: nginx
ports:
- containerPort: 80
name: web
volumeClaimTemplates:
- metadata:
name: nginx
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
```
Following the initial deployment, we have these objects:
```
❯ kubectl get statefulset,pod,pvc -ocustom-columns='KIND:.kind,NAME:.metadata.name,UID:.metadata.uid'
KIND NAME UID
StatefulSet nginx b1aa144d-3f16-448e-8e15-02d2c2d4b61a
Pod nginx-0 c00d97cc-39d7-4a95-b839-0910f911dbca
PersistentVolumeClaim nginx-nginx-0 2c624ff9-e856-4d2d-bfaf-527c6b770bc7
```
To provoke a replacement, we change the PVC template:
```
requests:
- storage: 1Gi
+ storage: 2Gi
```
Let's also increase the replicas:
```
spec:
- replicas: 1
+ replicas: 2
```
And deploy:
```
❯ pulumi up -f
Updating (dev)
Type Name Status Info
pulumi:pulumi:Stack issue-1831-dev 4 warnings; 2 messages
+- └─ kubernetes:apps/v1:StatefulSet nginx replaced (2s) [diff: ~spec]
Resources:
+-1 replaced
1 unchanged
```
Looking again at the objects:
```
❯ kubectl get statefulset,pod,pvc -ocustom-columns='KIND:.kind,NAME:.metadata.name,UID:.metadata.uid'
KIND NAME UID
StatefulSet nginx 135d9142-460c-4f64-82a6-8ce23427f52b
Pod nginx-0 c00d97cc-39d7-4a95-b839-0910f911dbca
Pod nginx-1 8c80932f-7051-4fc7-baeb-00dae4b07b64
PersistentVolumeClaim nginx-nginx-0 2c624ff9-e856-4d2d-bfaf-527c6b770bc7
PersistentVolumeClaim nginx-nginx-1 e4b4fd18-28b2-454b-8c6b-9fa06435d3d6
```
We see the expected result: the StatefulSet was replaced, the existing
pod was adopted, and a new pod was added w/ a PVC.
In more detail, the StatefulSet controller uses the selector to identify
existing pods, then chooses to delete or adopt based on suitability.
Note the updated owner reference on `nginx-0`:
```yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx-0
uid: c00d97cc-39d7-4a95-b839-0910f911dbca
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: StatefulSet
name: nginx
uid: 135d9142-460c-4f64-82a6-8ce23427f52b
```
To demonstrate how the StatefulSet controller might choose to delete the
existing pod rather than adopting it, let's change the image rather than
the replicas:
```
containers:
- - image: nginx:1.19.9
+ - image: nginx:1.19.10
requests:
- storage: 2Gi
+ storage: 3Gi
```
We deploy again and see that all pods were replaced.
```
KIND NAME UID
StatefulSet nginx ead53943-abc9-47c8-9393-326f845c7f42
Pod nginx-0 74752b8c-3979-478b-9be4-ff3ca1b0aa6f
Pod nginx-1 b6c2f0f6-f5ff-4e04-a1da-66966b8d697c
PersistentVolumeClaim nginx-nginx-0 2c624ff9-e856-4d2d-bfaf-527c6b770bc7
PersistentVolumeClaim nginx-nginx-1 e4b4fd18-28b2-454b-8c6b-9fa06435d3d6
```
Note that PVC `nginx-nginx-0` was not replaced and still has `storage:
1Gi`.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello!
Issue details
For certain statefulset update situations, pulumi needs to delete the statefulset and then create it again. One such case would be when modifying the storage size of a volume in the claim template. The issue here is that all of the statefulset pods will be terminated at once, and hence there won't be any pods to receive traffic.
K8s provides a
cascadeoption which allows it to delete the statefulset without it deleting the dependent pods. https://kubernetes.io/docs/tasks/run-application/delete-stateful-set/Currently I need to run manually kubectl with the cascade flag, pulumi refresh, and then pulumi up. Providing a way to configure the cascade option in code would allow Pulumi to delete the statefulset while keeping the pods, and then recreate it with the new changes.
Perhaps setting a new annotation on the statefulset would be the way to solve this?
This request has some relation to: #304
The text was updated successfully, but these errors were encountered: