Proposal for growing persistent volumes

[gnufied]

cc @kubernetes/sig-storage-pr-reviews @liggitt @eparis

[k8s-ci-robot]

@gnufied: These labels do not exist in this repository: sig/storage.

In response to this:

cc @kubernetes/sig-storage-pr-reviews @liggitt @eparis

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[thockin]

Isn't a resize specific to a filesystem? Do we expect all volume drivers to bundle all tools for all filesystems? That seems untenable.

[childsb]

This is really only for block storage and is similar to specifying the FS type of the PV and having kubelet format it for you.. And in that scenario we expect that the fs tools are on the node. Are we expecting anything besides XFS or EXT4 on block? For shared network storage types (ie gluster) this will be a noop.

The weird case may be azure or other Samba volumes. But either way I think the fs tools required is minimal. I would love it if the tools were containerized, but we dont require that now to format block PVs....

[gnufied]

I went ahead and had a look at various distributions and how they ship the file system utils, and one thing which I could confirm is if formatting utils for xfs or ext are available on the host - then resize utils are available too. I looked at - centos7, ubuntu 16.04 (server and minimal), Opensuse-Leap 42.2 .

On all these distros if you install formatting package (such xfsprogs) then it comes with resizing utility too.

I haven't confirmed this for all file system types and neither on all distros, so I wouldn't call my information conclusive. But - what I propose is, we start with supporting ext4 and xfs for resizing. That would cover majority of use cases. For more exhaustive solution - we may want to wait for containarization proposal on which @jsafrane is working on.

@thockin @childsb what you think? Is there any thing else we are missing?

[gnufied]

Also - we document the fact that, growing file system is only supported for ext4 and xfs in case of block device storage.

[thockin]

do those tools all have the same flags? Or do we need to open-code a list of supported filesystems and their tooling?

[gnufied]

@xingzhou I don't think a common utility for all volume plugins, but rather a common utility which can abstract away resize function(and possibly more file system operations) for all/most file system types.

[fabiand]

@aglitke Even if a pod is consuming the raw block device, I'd still assume that the host/kubelet is owning it - i.e. attaching/detaching it, and also rescanning it if necessary.
But if that's the case, the this could lead to surprises within the pod - if the device just becomes larger from one second to the other.

[gnufied]

@fabiand what kind of surprises you are talking about - if local device pod was using became larger? As proposed in this proposal - only device size will become larger. The size available to the pod will still be the same until file system has been resized too.

Also, I am not sure - if there is a way to centrally resize a device for local storage. It has to be done from daemonset (as per current local storage proposal) which actually does provisioning of local storage. All in all - I don't expect PV expand to be supported in local disks for while. What I would be worried about is - if current design somehow inhibits that, which I don't think it does. cc @msau42 for local storage bits.

[msau42]

For local storage, we're trying to put most of the logic in an external daemonset, to minimize kubelet dependencies, and to be able to support all the myriad ways to configure local storage, so each admin can customize it to their environment. So, if say someone wants to use lvm as their local storage backend, then they can have all the lvm tools in the daemonset and do the resizing from there. The daemonset needs some sort of signal from the system when to resize.

[fabiand]

@gnufied Right, that's true for volumes thus file-systems. But @aglitke was speaking of raw block volumes, and there the size change is - IIRC - known once we rescan the bus, at least in the case of (i)SCSI devices.

[thockin]

I dislike that the status is set in 2 different places - can we refine it to centralize that?

[gnufied]

pvc.status.capacity already exists, this proposal adds an additional field in PVs. but I will think if we can centralize this.

[gnufied]

I have removed this duplication now.

[saad-ali]

• How do we keep track of the status of a resize operation?
• What happens if the resize operation fails? How is that communicated to end user?
• What happens if the user resizes multiple times in rapid succession? If you prevent a resize operation from starting while an existing resize operation is pending how will you keep track of that? What if the user removes and readds the annotation?
• A subresource on the PVC might be a good API?

[gnufied]

@saad-ali my original plan was to add a field called pv.status.conditions []PvCondition which is similar to PodCondition. Something like:

type PvCondition {
  Type PvConditionType ("ResizeStarted|ResizeInProgress|ResizeComplete|ResizeFailed) 
  Status pvConditionStatus (True|False|Unknwon)
  Reason string
  Message string
}

type PersistentVolumeStatus {
  Condition []PvCondition
}

which would store different "status" of resize operation. The problem is - this feature being Alpha, we will have to either serialize this whole struct into a json string and then put it in annotation, or we will have to create new version of PersistentVolume apis.

For Alpha release:

• How do we keep track of the status of a resize operation?

We have alpha annotation volume.alpha.kuberenetes.io/fs-resize-pending and pv.status.capacity. Do we need additional information to track status of resize operation?

• What happens if the resize operation fails? How is that communicated to end user?

If cloudprovider resize failed, we are going to attach an event to the pvc. If file system resize failed - we are going to attach an event to the pod as well as pvc.

*What happens if the user resizes multiple times in rapid succession? If you prevent a resize operation from starting while an existing resize operation is pending how will you keep track of that? What if the user removes and readds the annotation?

This shouldn't be a problem - because file system resize is only performed when a new pod is started. So all rapid resizes are limited to cloudprovider api calls and they update pv.spec.capacity as expected. We don't stop users from doing resize when an existing resize is pending.

However if user manually edits the PV and removes the annotation - we do have a problem. Isn't this expected that users aren't supposed to touch kubernetes.io annotations?

A subresource on the PVC might be a good API?

I don't know what you mean. A subresource to represent what?

[thockin]

blech!! can we find a way orthogonalize filesystems and storage subsystems?

…

On Wed, May 24, 2017 at 2:46 PM, Hemant Kumar ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In contributors/design-proposals/grow-volume-size.md <#657 (comment)>: > + + In case where volume type requires no file system resize, both PV & PVC objects will be updated accordingly and `status.capacity` of both objects will reflect new size. + For volume plugins that require file system resize - an additional annotation called `volume.alpha.kubernetes.io/fs-resize-pending` <http://volume.alpha.kubernetes.io/fs-resize-pending> will be added to PV to communicate + to the Kubelet that File system must be resized when a new pod is started using the PV. + +* In case volume plugin doesn’t support resize feature. The resize API request will be rejected and PVC object will not be saved. This check will be performed via an admission controller plugin. + +* In case requested size is smaller than current size of PVC. A validation will be used to reject the API request. (This could be moved to admission controller plugin too.) + +* There will be additional checks in controller that grows PV size - to ensure that we do not make cloudprovider API calls that can reduce size of PV. + +* To consider cases of missed PVC update events, an additional loop will reconcile bound PVCs with PVs. + +* Resource Quota code in admission controller has to be updated to consider PVC updates. + +* The resize of file system will be performed on kubelet. If there is a running pod - no operation will be performed. Only when a new pod is started using same PVC - then kubelet will match device size and size of pv and attempt a resize of file system. resizing filesystem will be a volume plugin function. It is upto volume plugin maintainer to correctly implement this. In following cases no resize will be necessary and hence volume plugin can return success without actually doing anything. That is somewhat problematic thing about resize(when compared to mkfs). Resize command for each file system is going to be different and their flags too. So, we will have to pretty much compose appropriate command via switch statement or something like that (or best case - define some interface functions) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#657 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVMHOskkxOP2fatLRTIm6bxb7W0PSks5r9KUqgaJpZM4NkQSP> .

[gnufied]

hmm - orthogonalize at what level? There are two distinct problems:

1. Having to open-code support for each file system (or at least ext4 and xfs to begin with). We can mitigate some of this by - abstracting away the whole file system tooling in a library. So as a call from k8s boils down to:

fs := NewFileSystem("xfs")
fs.resize(device_id, "100GB")
fs.format(device_id)

But ultimately this library has to hide that detail. We can implement this library as part of this proposal, so as support for different file systems can be expanded in k8s without possibly changing k8s core.

2. Unavailability of tools on host. That is something we can't easily solve without containarizing all the tools.

[taherv]

flexvolume seems to be missing from this table .... is that intentional ? What about flexvolume with respect to this proposal ?

[gnufied]

If flex api adds support for a resize call - it can be added to the list. But for now - since it doesn't, I haven't added flex to the list.

[xingzhou]

so if user uses LVM as storage pool, the resize operation will be rejected in this case? do we need another feature request for FlexVolume to add resize call support?

[taherv]

+1 for including flexvol support (either in this proposal, or as a sister-proposal).

[xingzhou]

there are some requirements in supporting LVM from our side, I can help create a proposal on FlexVolume for this and take care of the development, what's your idea @gnufied?

[taherv]

i can be an extra hand for flexvol support, too, @xingzhou

[gnufied]

Yeah, I think we have to have another proposal for resize of LVM volumes via Flex. It wouldn't hurt to start a parallel proposal for flex that implements resizing based on this proposal. cc @chakri-nelluri

[xingzhou]

Sure, let me prepare and submit a new proposal on this, thx
/cc @taherv

[jsafrane]

There is /usr/sbin/fsadm which can resize ext2/3/4, xfs and RaiserFS. It comes as part of LVM utilities and I don't know if it's available in "minimal" distros.

[j-griffith]

Extend isn't horrible by any stretch. I'd prefer we call it "extend" however to remove any ambiguity, and to stress that there's no interest (I hope) in reduce.

I'd certainly consider this much lower priority than some of the provisioning work we've been talking about but maybe that's just me.

[npramod]

How about storage provisioned by out-of-tree provisioners that support Dynamic Provisioning(DP) today? Any thoughts on how resize would work there. For legacy/non-cloud native storage systems, Storage subsystem side operations might be required before a host side filesystem resize.

[gnufied]

Any external controller can implement resize mechanism of cloudprovider and leave host side file system resize to kubelet. Correct me if I am wrong but I don't think - this proposal stops external provisioners from doing that.

[ddysher]

My understanding is that external controller can work like in-tree controller by just watching update event and resize volume accordingly, but we do need to clearly outline the 'protocol' we used (e.g. volume.alpha.kubernetes.io/fs-resize-pending annotation).

[gnufied]

Yep that is correct. I think as long as protocol being specified here is followed, any external controller can perform resize. This shouldn't be a problem.

[yanivlavi]

This sounds like what I asked for below.
Don't we want to have the ability to resize the FS on external resize as well? Reactive fs resize.

[gnufied]

As long as - necessary fields/annotation that exist on PV that tells kubelet that a fs resize is needed - Kubelet will perform fs resize. Ideally - we don't want admins to modify those annotations/fields manually - but we are not going to put additional restrictions if they want to!

So in a sense, if admin has resized a volume manually (via ebs resize command) and did not also update the PV object with new size and volume.alpha.kubernetes.io/fs-resize-pending annotation then no FS resize will be performed on kubelet. But if admin manually edits the PV and makes same changes that resize controller would do, then we can't distinguish manual changes from resize controller change and hence a FS resize will be performed. But just because we can, doesn't mean we should.

[yanivlavi]

I feel this is no standard. Similar to ISCSI or NFS resizing.
It is expected that volume size metadata is updated (that is why there are events), even if the FS is not resized on top.

[gnufied]

It is expected that volume size metadata is updated (that is why there are events), even if the FS is not resized on top.

I am sorry, I don't understand what that means. Can you elaborate? May be perhaps with some examples.

[yanivlavi]

Would the size update even if the change was done outside Kube?
If the admin did the change for the user via the storage management API.

[gnufied]

If admin modifies volume outside kube, as per this design - size update will not be observed by Kubernetes. I would argue that - it is conscious decision that we have taken, if admin modifies volume objects outside, he is entirely responsible for resizing file systems or making corresponding changes in volume objects.

[yanivlavi]

If the service admin makes a change by request for a volume I argue this should be detected.
Maybe the FS resize should be manually triggered, but the sizing should reflect the storage service state, not make it inconsistent with Kube.
The storage service admin might not care for specifically Kube as a application and would expect it to be updated automatically like in dealing with OSes using volumes.

[ddysher]

Why is this a problem, I think there is quite a few features using json string?

The problem is - this feature being Alpha, we will have to either serialize this whole struct into a json string and then put it in annotation, or we will have to create new version of PersistentVolume apis.

@gnufied
I'm interested in extending pv/pvc to have some sort of health status report by adding taints or conditions to PVs, so there might be some overlap related to PV conditions described here. For example, if a PV becomes unhealthy (e.g. lost connection, local volume partition error), then we'll want to taint the PV and only PVCs tolerate the taint can use it; for already bound PV/PVC, we'll do an unbind and reschedule PVC.

For node case, kubelet reports condition and node controller taints the node. For volume, maybe we can have volume plugin reports the condition and pv controller to taint the node, is this align with ur plan?

[gnufied]

then we'll want to taint the PV and only PVCs tolerate the taint can use it; for already bound PV/PVC, we'll do an unbind and reschedule PVC.

What you mean by reschedule the PVC? You mean unbind from current PV and bind it to new one? We have to be careful because - PVs usually contain data and say if a MySQL PVC is bound to a new PV with no data in it, we have problem.

[ddysher]

Yes, unbind PVC from current PVC. Starting from v1.8, scheduling will take PV/PVC into consideration (now they are two separate process).

Agree we should be careful about this. So like node NotReady taint, we might add tolerations to pods in admission controller by default, unless user explicitly says otherwise, in which case, we can freely do an unbind.

[gnufied]

@saad-ali @thockin PTAL. I have flexed out the details of the proposal quite a bit. Improved error handling.

[gnufied]

also @liggitt let me know if everything looks good from #api-machinery and authorization perspective.

[ddysher]

What are the conditions for unbound or pending PVCs? are we simply ignore the new field or we set everything to False, i.e. Ready=False, ResizeStarted=False, ResizeFailed=False.

[gnufied]

I am thinking Conditions for pending or unbound pvcs would be simply empty. But pending in this context generally means pending phase.

[ddysher]

Is this true for file system resize as well? My understanding is that kubelet will observe pv.spec.capacity greater than pvc.status.spec.capacity and perform a file system resize.

[gnufied]

Yeah that is a good point, I think we can't avoid doing file system resize. I do not see any particular harm in doing so too.

[jsafrane]

Mising a note about external provisioners - perhaps the (internal) admission controller should ignore those and let the external provisioner implement its own (external) admission service.

[gnufied]

ack

[k8s-github-robot]

Automatic merge from submit-queue