Implement Generic Batching Framework + Batch DescribeVolume API requests

[torredil]

Is this a bug fix or adding new feature?

New feature.

What is this PR about? / Why do we need it?

This PR implements a generic batching framework, designed to facilitate the aggregation and execution of tasks -- most notably batched API requests for DescribeVolume -- used by the driver to poll for volume state and determine attachment status.

Today, the driver makes one DV API call per volume, however, DV can return up to 500 results. These changes allow for utilizing the API's capacity to handle up to 500 volumes at once, which significantly helps mitigate rate limit issues with DV and enhance performance when using the driver under heavy load scenarios as a result.

There are two parts to this PR, separated by commits:

1. Generic Batching Framework.

2. Batch DescribeVolume API Requests.

What testing is done?

• make test
• CI
• Manual testing

[torredil]

Kustomize CI job needs to be updated. Driver fails to enter a running state because --batching is passed into the controller container for driver version 1.24 (need to deploy driver using image built for CI instead).

[torredil]

Re Kustomize CI job needs to be updated -- we decided to run pull-aws-ebs-csi-driver-external-test-kustomize only on release branches. This is better than doing in-place modifications of the image repository / tag values.

An additional advantage in doing the above is that we implicitly validate image promotion for new releases moving forward since we pull from the official public ECR gallery.

[torredil]

/hold will retest once changes to test-infra have been merged.

[torredil]

/retest

[ConnorJC3]

/close

[k8s-ci-robot]

@ConnorJC3: Closed this PR.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ConnorJC3]

/reopen

[k8s-ci-robot]

@ConnorJC3: Reopened this PR.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[torredil]

/unhold

[ConnorJC3]

Needs rebase, one comment, otherwise lgtm

[AndrewSirenko]

/lgtm
(Minus the cloud_test merge conflict and 1 question)

[ConnorJC3]

/lgtm
(reapplying @AndrewSirenko's lgtm)

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ConnorJC3

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ConnorJC3]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[torredil]

/retest

[torredil]

For peace of mind, investigated the resizing failure in pull-aws-ebs-csi-driver-e2e-single-az:

1 driver.go:125] "GRPC error" err="rpc error: code = Internal desc = Could not resize volume \"vol-003bbe6304464240d\": rpc error: code = Internal desc = Could not modify volume \"vol-003bbe6304464240d\": volume \"vol-003bbe6304464240d\" in OPTIMIZING state, cannot currently modify"

In conclusion, the error observed above is unrelated to this PR and occurs when a volume takes longer than 1 minute to transition from an optimizing state to completed. See a prior example here. This behavior can be easily reproduced outside of the driver by simply modifying a volume via the CLI or console.

Based on my observation, it is fairly common to see a volume remain in optimizing 99% for up to 5 minutes before transitioning to complete. However, the single AZ tests expect resizing to succeed within one minute:

Nov  7 17:54:03.198: INFO: Unexpected error: 
  <*errors.errorString | 0xc000a9b9a0>: 
    Gave up after waiting 1m0s for pv "pvc-af4167dc-d33b-4b78-a3f1-35d6e2856d2d" to complete resizing
  {
       s: "Gave up after waiting 1m0s for pv \"pvc-af4167dc-d33b-4b78-a3f1-35d6e2856d2d\" to complete resizing",
  }

Given that we recently introduced more tests that exercise this code path in single-az, we are more likely to observe the error moving forward if that timeout value is not relaxed.

/retest

[k8s-ci-robot]

@torredil: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-aws-ebs-csi-driver-external-test-kustomize	c1a894c	link	false	/test pull-aws-ebs-csi-driver-external-test-kustomize

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[torredil]

ResizeOrModifyDisk needs to be revisited. There is a discrepancy in the logic.

According to cloud, a modification is considered done if the volume is an optimizing state:

aws-ebs-csi-driver/pkg/cloud/cloud.go

Lines 1320 to 1323 in d071db8

	func volumeModificationDone(state string) bool {
	if state == ec2.VolumeModificationStateCompleted || state == ec2.VolumeModificationStateOptimizing {
	return true
	}

In the case where ResizeOrModifyDisk hasn't been previously called for a volume, we simply move on to check if the volume is in the desired state. If it is, ResizeOrModifyDisk succeeds even if the volume is still optimizing. However, if checkDesiredState returns an error or the sidecar times out, latestMod won't be nil the second time around and we error out if the volume is optimizing:

aws-ebs-csi-driver/pkg/cloud/cloud.go

Lines 1298 to 1300 in d071db8

	} else if state == ec2.VolumeModificationStateOptimizing {
	return true, 0, fmt.Errorf("volume %q in OPTIMIZING state, cannot currently modify", volumeID)
	}

[torredil]

/retest
/hold

[torredil]

/unhold