Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for io2 Block Express volumes #1409

Merged
merged 1 commit into from
Oct 4, 2022
Merged

Add support for io2 Block Express volumes #1409

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 18 additions & 16 deletions docs/parameters.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,25 +3,27 @@ There are several optional parameters that may be passed into `CreateVolumeReque

The AWS EBS CSI Driver supports [tagging](tagging.md) through `StorageClass.parameters` (in v1.6.0 and later).

| Parameters | Values | Default | Description |
|-----------------------------|----------------------------------------------------------------|----------|---------------------|
| "csi.storage.k8s.io/fstype" | xfs, ext2, ext3, ext4 | ext4 | File system type that will be formatted during volume creation. This parameter is case sensitive! |
| "type" | io1, io2, gp2, gp3, sc1, st1, standard, sbp1, sbg1 | gp3* | EBS volume type. |
| "iopsPerGB" | | | I/O operations per second per GiB. Can be specified for IO1, IO2, and GP3 volumes. |
| "allowAutoIOPSPerGBIncrease"| true, false | false | When `"true"`, the CSI driver increases IOPS for a volume when `iopsPerGB * <volume size>` is too low to fit into IOPS range supported by AWS. This allows dynamic provisioning to always succeed, even when user specifies too small PVC capacity or `iopsPerGB` value. On the other hand, it may introduce additional costs, as such volumes have higher IOPS than requested in `iopsPerGB`.|
| "iops" | | | I/O operations per second. Can be specified for IO1, IO2, and GP3 volumes. |
| "throughput" | | 125 | Throughput in MiB/s. Only effective when gp3 volume type is specified. If empty, it will set to 125MiB/s as documented [here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html). |
| "encrypted" | | | Whether the volume should be encrypted or not. Valid values are "true" or "false". |
| "kmsKeyId" | | | The full ARN of the key to use when encrypting the volume. If not specified, AWS will use the default KMS key for the region the volume is in. This will be an auto-generated key called `/aws/ebs` if not changed. |
| Parameters | Values | Default | Description |
|------------------------------|----------------------------------------------------|---------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| "csi.storage.k8s.io/fstype" | xfs, ext2, ext3, ext4 | ext4 | File system type that will be formatted during volume creation. This parameter is case sensitive! |
| "type" | io1, io2, gp2, gp3, sc1, st1, standard, sbp1, sbg1 | gp3* | EBS volume type. |
| "iopsPerGB" | | | I/O operations per second per GiB. Can be specified for IO1, IO2, and GP3 volumes. |
| "allowAutoIOPSPerGBIncrease" | true, false | false | When `"true"`, the CSI driver increases IOPS for a volume when `iopsPerGB * <volume size>` is too low to fit into IOPS range supported by AWS. This allows dynamic provisioning to always succeed, even when user specifies too small PVC capacity or `iopsPerGB` value. On the other hand, it may introduce additional costs, as such volumes have higher IOPS than requested in `iopsPerGB`. |
| "iops" | | | I/O operations per second. Can be specified for IO1, IO2, and GP3 volumes. |
| "throughput" | | 125 | Throughput in MiB/s. Only effective when gp3 volume type is specified. If empty, it will set to 125MiB/s as documented [here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html). |
| "encrypted" | true, false | false | Whether the volume should be encrypted or not. Valid values are "true" or "false". |
| "blockExpress" | true, false | false | Enables the creation of [io2 Block Express volumes](https://aws.amazon.com/ebs/provisioned-iops/#Introducing_io2_Block_Express) by increasing the IOPS limit for io2 volumes to 256000. Volumes created with more than 64000 IOPS will fail to mount on instances that do not support io2 Block Express. |
| "kmsKeyId" | | | The full ARN of the key to use when encrypting the volume. If not specified, AWS will use the default KMS key for the region the volume is in. This will be an auto-generated key called `/aws/ebs` if not changed. |

**Appendix**
* `gp3` is currently not supported on outposts. Outpost customers need to use a different type for their volumes.
* Unless explicitly noted, all parameters are case insensitive (e.g. "kmsKeyId", "kmskeyid" and any other combination of upper/lowercase characters can be used).
* If the requested IOPs (either directly from `iops` or from `iopsPerGB` multiplied by the volume's capacity) produces a value above the maximum IOPs allowed for the [volume type](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html), the IOPS will be capped at the maximum value allowed. If the value is lower than the minimal supported IOPS value per volume, either an error is returned (the default behavior), or the value is increased to fit into the supported range when `allowautoiopspergbincrease` is `"true"`.
* If the requested IOPS (either directly from `iops` or from `iopsPerGB` multiplied by the volume's capacity) produces a value above the maximum IOPS allowed for the [volume type](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html), the IOPS will be capped at the maximum value allowed. If the value is lower than the minimal supported IOPS value per volume, either an error is returned (the default behavior), or the value is increased to fit into the supported range when `allowautoiopspergbincrease` is `"true"`.
* You may specify either the "iops" or "iopsPerGb" parameters, not both. Specifying both parameters will result in an invalid StorageClass.

| Volume Type | Min total IOPS | Max total IOPS | Max IOPS per GB |
|-----------------------------|----------------------------------------|------------------|-------------------|
| IO1 | 100 | 64000 | 50 |
| IO2 | 100 | 64000 | 500 |
| GP3 | 3000 | 16000 | 500 |
| Volume Type | Min total IOPS | Max total IOPS | Max IOPS per GB |
|----------------------------|----------------|---------------|-------------------|
| io1 | 100 | 64000 | 50 |
| io2 (blockExpress = false) | 100 | 64000 | 500 |
| io2 (blockExpress = true) | 100 | 256000 | 500 |
| gp3 | 3000 | 16000 | 500 |
26 changes: 16 additions & 10 deletions pkg/cloud/cloud.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,15 +63,16 @@ const (
// AWS provisioning limits.
// Source: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html
const (
io1MinTotalIOPS = 100
io1MaxTotalIOPS = 64000
io1MaxIOPSPerGB = 50
io2MinTotalIOPS = 100
io2MaxTotalIOPS = 64000
io2MaxIOPSPerGB = 500
gp3MaxTotalIOPS = 16000
gp3MinTotalIOPS = 3000
gp3MaxIOPSPerGB = 500
io1MinTotalIOPS = 100
io1MaxTotalIOPS = 64000
io1MaxIOPSPerGB = 50
io2MinTotalIOPS = 100
io2MaxTotalIOPS = 64000
io2BlockExpressMaxTotalIOPS = 256000
io2MaxIOPSPerGB = 500
gp3MaxTotalIOPS = 16000
gp3MinTotalIOPS = 3000
gp3MaxIOPSPerGB = 500
)

var (
Expand Down Expand Up @@ -192,6 +193,7 @@ type DiskOptions struct {
AvailabilityZone string
OutpostArn string
Encrypted bool
BlockExpress bool
// KmsKeyID represents a fully qualified resource name to the key to use for encryption.
// example: arn:aws:kms:us-east-1:012345678910:key/abcd1234-a123-456a-a12b-a123b4cd56ef
KmsKeyID string
Expand Down Expand Up @@ -316,7 +318,11 @@ func (c *cloud) CreateDisk(ctx context.Context, volumeName string, diskOptions *
minIops = io1MinTotalIOPS
maxIopsPerGb = io1MaxIOPSPerGB
case VolumeTypeIO2:
maxIops = io2MaxTotalIOPS
if diskOptions.BlockExpress {
maxIops = io2BlockExpressMaxTotalIOPS
} else {
maxIops = io2MaxTotalIOPS
}
minIops = io2MinTotalIOPS
maxIopsPerGb = io2MaxIOPSPerGB
case VolumeTypeGP3:
Expand Down
20 changes: 20 additions & 0 deletions pkg/cloud/cloud_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -496,6 +496,26 @@ func TestCreateDisk(t *testing.T) {
},
expErr: nil,
},
{
name: "success: large io2 Block Express with too high iopsPerGB",
volumeName: "vol-test-name",
diskOptions: &DiskOptions{
CapacityBytes: util.GiBToBytes(3333),
Tags: map[string]string{VolumeNameTagKey: "vol-test", AwsEbsDriverTagKey: "true"},
VolumeType: VolumeTypeIO2,
IOPSPerGB: 100000,
BlockExpress: true,
},
expDisk: &Disk{
VolumeID: "vol-test",
CapacityGiB: 3333,
AvailabilityZone: defaultZone,
},
expCreateVolumeInput: &ec2.CreateVolumeInput{
Iops: aws.Int64(256000),
},
expErr: nil,
},
{
name: "success: create volume when zone is snow and add tags",
volumeName: "vol-test-name",
Expand Down
3 changes: 3 additions & 0 deletions pkg/driver/constants.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,9 @@ const (
// provisioned volume
PVNameKey = "csi.storage.k8s.io/pv/name"

// BlockExpressKey increases the iops limit for io2 volumes to the block express limit
BlockExpressKey = "blockExpress"

// TagKeyPrefix contains the prefix of a volume parameter that designates it as
// a tag to be attached to the resource
TagKeyPrefix = "tagSpecification"
Expand Down
12 changes: 10 additions & 2 deletions pkg/driver/controller.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ func (d *controllerService) CreateVolume(ctx context.Context, req *csi.CreateVol
iops int
throughput int
isEncrypted bool
blockExpress bool
kmsKeyID string
scTags []string
volumeTags = map[string]string{
Expand Down Expand Up @@ -160,8 +161,6 @@ func (d *controllerService) CreateVolume(ctx context.Context, req *csi.CreateVol
case EncryptedKey:
if value == "true" {
isEncrypted = true
} else {
isEncrypted = false
}
case KmsKeyIDKey:
kmsKeyID = value
Expand All @@ -174,6 +173,10 @@ func (d *controllerService) CreateVolume(ctx context.Context, req *csi.CreateVol
case PVNameKey:
volumeTags[PVNameTag] = value
tProps.PVName = value
case BlockExpressKey:
if value == "true" {
blockExpress = true
}
default:
if strings.HasPrefix(key, TagKeyPrefix) {
scTags = append(scTags, value)
Expand All @@ -189,6 +192,10 @@ func (d *controllerService) CreateVolume(ctx context.Context, req *csi.CreateVol
}
}

if blockExpress && volumeType != cloud.VolumeTypeIO2 {
return nil, status.Errorf(codes.InvalidArgument, "Block Express is only supported on io2 volumes")
}

snapshotID := ""
volumeSource := req.GetVolumeContentSource()
if volumeSource != nil {
Expand Down Expand Up @@ -241,6 +248,7 @@ func (d *controllerService) CreateVolume(ctx context.Context, req *csi.CreateVol
AvailabilityZone: zone,
OutpostArn: outpostArn,
Encrypted: isEncrypted,
BlockExpress: blockExpress,
KmsKeyID: kmsKeyID,
SnapshotID: snapshotID,
}
Expand Down