Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

document pvc finalizer issue during 1.10 -> 1.9 downgrade #7731

Closed
wants to merge 5 commits into from
Closed

document pvc finalizer issue during 1.10 -> 1.9 downgrade #7731

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
e16c2d1
document pvc finalizer issue during 1.10 -> 1.9 downgrade
rootfs Mar 13, 2018
c94778b
feedback addressed
rootfs Mar 14, 2018
a1edd39
change the flow: allow downgrade first but deal with finalizer after …
rootfs Mar 16, 2018
47f3147
be more specific about finalizer element to remove
rootfs Mar 19, 2018
df9d49f
Update pvc-finalizer-downgrade-issue.md
Bradamant3 Mar 21, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions docs/reference/pvc-finalizer-downgrade-issue.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
* TOC
{:toc}

---
title: Kubernetes Downgrade issue from 1.10 to 1.9 due to PV/PVC Protection
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's be more precise here:
"Issue downgrading Kubernetes from 1.10 to 1.9 if StorageObjectInUseProtection admission controller is enabled"

I suggest the change because of the name of the flag the user actually sets. I understand that the finalizer names are different, and the user needs to know what they are to troubleshoot.

The docs also explain about the finalizers, but wouldn't a user be more likely to think in terms of the plugin name?

If I have time, I'll submit a PR and flag y'all on it, to explain the feature more clearly in the docs, too. It took me a number of reads to sort out what was going on, and I know about admission controllers and finalizers. (I am also in high end-of-release-cycle mode, I admit.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Bradamant3 yes your rephrase sounds good to me. I feel both doc (admission controller in admin doc in the reference) and this one need to be in sync to help end users know what the term is and what it is for.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would vote for a shorter title for this because when the article is online, long titles would be a problem for navigation.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The feature is called StorageObjectInUseProtection in K8s 1.10 and PVCProtection in K8s 1.9 that's why I would change the title in such a way that it refers to the feature names.
Do we need to include exact numbers from 1.10 to 1.9 in the title?
What about title: StorageObjectInUse/PVC Protection Downgrade Issue?

---

## PV/PVC Protection in Kubernetes 1.10
Copy link
Contributor

@Bradamant3 Bradamant3 Mar 14, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To work with previous edit:
"Storage Object in Use Protection in Kubernetes 1.10"


When enabled, [PV/PVC Protection](docs/admin/admission-controllers.md#storage-object-in-use-protection-beta) prevents PV/PVC from being removed when the finalizers are removed.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"If you enable the admission controller StorageObjectInUseProtection, your PersistentVolume and PersistentVolumeClaim objects are not removed if the related pv-protection and pvc-protection finalizers are removed."

Not sure I've got this right because there seems to be some contradiction between the text ^^ and text below.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch, it is should be PersistentVolume and PersistentVolumeClaim objects are not removed if the related pv-protection and pvc-protection finalizers are still *present*

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch, it is should be PersistentVolume and PersistentVolumeClaim objects are not removed if the related pv-protection and pvc-protection finalizers are still present

Correct, when the finalizer is present in an PV or PVC object, the object is not removed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The link should be /docs/admin/admission-controllers.md#storage-object-in-use-protection-beta, otherwise it will be broken. NB the first slash.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because the feature is called StorageObjectInUseProtection in K8s 1.10 I would:

s/[PV/PVC Protection]/[`StorageObjectInUseProtection`]


## Downgrading issue

After downgrading from Kubernetes 1.10 to 1.9, PV/PVCs that are created in Kubernetes 1.10 with PVC Protection cannot be removed.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"After you downgrade from Kubernetes 1.10 to 1.9, PersistentVolume or PersistentVolumeClaim objects that were created with version 1.10 cannot be removed. This is because their finalizers are not recognized in version 1.9."

I'm extrapolating from the first section -- is this indeed correct? Clearer to be explicit even if it looks repetitive to someone who knows the feature.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that is the case, let @pospispa confirm

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, let me firstly explain the purpose of the StorageObjectInUseProtection feature.

When the StorageObjectInUseProtection feature is disabled PVs and PVCs have empty finalizers, i.e. are not protected. When a user deletes a PVC (no finalizer present) that is in active use by a pod the PVC is removed immediately, therefore, the user may loose data. Similarly, when an admin deletes a PV (no finalizer present) that is in active use (i.e. bound to a PVC that is in active use) data may be lost again.

So when the StorageObjectInUseProtection feature is enabled finalizers are added to PVs and PVCs immediately when they are created (this is done by the StorageObjectInUseProtection admission plugin) so these PVs and PVCs are now protected. Now, when the PV or PVC is deleted it is not removed immediately, they transition into Terminating phase. PVs or PVCs in Terminating phase are removed after it's finalizer is removed. In K8s 1.10 there are PV and PVC Protection controllers that remove finalizers automatically when a PV or PVC in Terminating phase is not in active use.

In K8s 1.9 the finalizers work in the same way as in K8s 1.10, i.e. when a finalizer is present in an object and the object is deleted the object is not removed immediately, but the object transitions into Terminating phase. The object is removed after it's finalizer is removed.

Note: there is PVCProtection alpha feature in K8s 1.9. When this feature is enabled PVCProtection controller automatically removes finalizers from PVCs when they are both in Terminating phase and not in active use by a pod. However, there's no controller that would remove finalizers from PVs.

I personally would describe the downgrading issue in the below way:

After downgrading from Kubernetes 1.10 to 1.9, PV/PVCs that contain a finalizer cannot be removed until their finalizer is removed.


## Workaround

Currently PV/PVC finalizers have to be manually removed so PV/PVC can be removed after downgrading to Kubernetes 1.9.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Currently the pv-protection and pvc-protection finalizers must be removed manually before you downgrade so that PVs and PVCs can be removed after you downgrade to version 1.9. Here's what to do:"

yeah, I added "before you downgrade" because that's what it looks like from the following instructions, but it was not completely clear. Also adding another step to make when you do what even clearer.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good to me


Before downgrading to Kubernetes 1.9, disable `StorageObjectInUseProtection` plugin and restart admission controller.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"1. Before you downgrade, disable the StorageObjectInUseProtection plugin and restart the admission controller."

Can we also please provide the shell command here? (and if you indent four spaces it will keep the step numbering, with my great thanks)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hyperkube apiserver --disable-admission-plugins=StorageObjectInUseProtection ... (rest of the options omitted)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or kube-apiserver --disableadmissionplugins=StorageObjectInUseProtection ...? (to match what's currently in the docs)

Also -- I see nothing in the admission controller docs about a restart. If it's necessary, what's the command? Do you have to restart the apiserver?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--disable-admission-plugins is the command line option, there is - in between words.

The restart process is to stop apiserver, and start it again but with a different command line option as above:

  • if apiserver is managed by systemd, stop it using systemd stop <apiserver service name>. Modify apiserver service unit service file, reload the service, and start it using systemd start command
  • if apiserver is started through command line, kill the process and run the command again with the above command line option

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How to disable StorageObjectInUseProtection plugin and restart admission controller? I don't read the details of the doc PV/PVC Protection referenced above. If the approach is given in the doc, it'd be good to mention this. If not, it'd be great to give some details or reference some doc here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just notice that @Bradamant3 already mentioned this in #7731 (comment) :)


Then patch PV/PVC (e.g. `pv1`) using the following command:

```bash
kubectl patch pv pv1 --type=json -p='[{"op": "remove", "path": "/metadata/finalizers"}]'
```

then verify the finalizers are removed:
```bash
kubectl get pv pv1 -o yaml |grep finalizer
# (result should be empty)
```
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And for ll22 to end:

1. Patch the PV or PVC, as in the following command, where `pv1` is the name of the PV to patch:

     ```bash
    kubectl patch pv pv1 --type=json -p='[{"op": "remove", "path": "/metadata/finalizers"}]'
    ````

1. Verify the finalizers are removed:

    ```bash
    kubectl get pv pv1 -o yaml |grep finalizer
    ```

    The result should be empty.

1. You can now safely downgrade to version 1.9.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds good. this process has to iterate over all pv/pvcs created in 1.10