Add support for io2 Block Express volumes

[ConnorJC3]

Signed-off-by: Connor Catlett [email protected]

[ConnorJC3]

/hold for reviews

Me and @torredil manually tested, this is good for review.

[torredil]

$ kubectl describe node i-0a8088f85b6d3bedb          
                             
Name:               i-0a8088f85b6d3bedb
Roles:              node
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=r5b.large
                    node.kubernetes.io/instance-type=r5b.large

$ kubectl get pvc           
                           
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
ebs-claim   Bound    pvc-862780bc-9b3b-4abb-8c3f-6f3c303a64d2   2000Gi     RWO            ebs-sc         2m51s

$ cat storageclass.yaml   
                                                           
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-sc
parameters:
  type: io2
  iops: "100000"
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer

$ cat claim.yaml      
                                                            
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-claim
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ebs-sc
  resources:
    requests:
      storage: 2000Gi

/lgtm
/hold

[torredil]

cc: @rdpsin

[rdpsin]

/lgtm

Can we also confirm that setting IOPS greater than 64K on non-bx instances will return an error? (Tbf, I don't see why not, but would still like to confirm).

[rdpsin]

FYI, it'd be useful to add some details in the commit message for posterity.

[rdpsin]

Thanks.

/lgtm

[ConnorJC3]

@rdpsin there's no way to know at the time of creation what type of instance the volume will be attached to. so it will be created, but fail to attach to instances that do not support block express.

Unfortunately the error message is somewhat poor (Could not attach volume "vol-006100676decdeaa0" to node "i-01365b727b32e9f74": attachment of disk "vol-006100676decdeaa0" failed, expected device to be attached but was detaching), but I believe that is something that has to be improved on the EBS side.

[rdpsin]

I see. This can potentially lead to leaked resources. As long as we are explicit in our docs, I guess it should be fine. Can you add this?

[torredil]

Discussed offline and decided against instance type validation using metadata during volume mounting to:

1. Avoid introducing new dependency on metadata. Long-term, we want to move to a world where metadata is an entirely optional dependency and not a requirement on both the controller & node.
2. Avoid being fragile to changes by requiring a CSI driver update when/if EBS supports block express on other instance types. It would be best to rely on the EBS api as source of truth. The downside here is that setting IOPS > 64K on non-bx supported instances results in failed, expected device to be attached but was detaching which is not very helpful.

Manually tested, these changes lgtm.

[rdpsin]

/lgtm

[torredil]

/unhold

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ConnorJC3, torredil

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ConnorJC3,torredil]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment