Parameters on this idempotent request are inconsistent with parameters used in previous request

[ananthbh-te]

/kind bug

**When i try to create a new volume with EBS csi driver installed, i'm facing this issue.

This was not the case before installing EBS CSI driver. **

EBS csi driver should create the volume

---

PVC events

Events:
  Type     Reason              Age   From                                                                                      Message
  ----     ------              ----  ----                                                                                      -------
  Warning  ProvisioningFailed  68s   ebs.csi.aws.com_ebs-csi-controller-847d68f976-tjn7h_a6ba8172-6d47-4c2d-a5a0-e52b7bd611af  failed to provision volume with StorageClass "influxdb-data-kube-services-pvc-class": rpc error: code = Internal desc = Could not create volume "pvc-d34ce440-7161-4a88-8363-001a057eca81": failed to get an available volume in EC2: InvalidVolume.NotFound: The volume 'vol-086af10dba8b4b9be' does not exist.
           status code: 400, request id: bf656776-28fa-469b-9ae1-248725b0a006
  Normal   Provisioning          4s (x7 over 72s)  ebs.csi.aws.com_ebs-csi-controller-847d68f976-tjn7h_a6ba8172-6d47-4c2d-a5a0-e52b7bd611af  External provisioner is provisioning volume for claim "kube-services/influxdb-data"
  Warning  ProvisioningFailed    4s (x6 over 67s)  ebs.csi.aws.com_ebs-csi-controller-847d68f976-tjn7h_a6ba8172-6d47-4c2d-a5a0-e52b7bd611af  failed to provision volume with StorageClass "influxdb-data-kube-services-pvc-class": rpc error: code = AlreadyExists desc = Could not create volume "pvc-d34ce440-7161-4a88-8363-001a057eca81": Parameters on this idempotent request are inconsistent with parameters used in previous request(s)
  Normal   ExternalProvisioning  3s (x7 over 72s)  persistentvolume-controller                                                               waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator

logs of ebs-csi-controller pod

kubectl logs ebs-csi-controller-847d68f976-swkpq -n kube-system
Defaulted container "ebs-plugin" out of: ebs-plugin, csi-provisioner, csi-attacher, csi-snapshotter, csi-resizer, liveness-probe
I1108 18:21:18.713260       1 driver.go:75] "Driver Information" Driver="ebs.csi.aws.com" Version="v1.21.0"
ANANTHBH-M-9N44:cloud-components ananthbh$ kubectl logs ebs-csi-controller-847d68f976-tjn7h -n kube-system
Defaulted container "ebs-plugin" out of: ebs-plugin, csi-provisioner, csi-attacher, csi-snapshotter, csi-resizer, liveness-probe
I0201 06:29:22.915609       1 driver.go:75] "Driver Information" Driver="ebs.csi.aws.com" Version="v1.21.0"
I0201 06:29:46.561742       1 controller.go:146] "\"fstype\" is deprecated, please use \"csi.storage.k8s.io/fstype\" instead"
E0201 06:29:46.804122       1 driver.go:122] "GRPC error" err="rpc error: code = AlreadyExists desc = Could not create volume \"pvc-89862b9e-f96a-40c0-9176-9986c7d37b7a\": Parameters on this idempotent request are inconsistent with parameters used in previous request(s)"
I0201 06:29:47.806216       1 controller.go:146] "\"fstype\" is deprecated, please use \"csi.storage.k8s.io/fstype\" instead"
E0201 06:29:48.003343       1 driver.go:122] "GRPC error" err="rpc error: code = AlreadyExists desc = Could not create volume \"pvc-89862b9e-f96a-40c0-9176-9986c7d37b7a\": Parameters on this idempotent request are inconsistent with parameters used in previous request(s)"
I0201 06:29:50.004655       1 controller.go:146] "\"fstype\" is deprecated, please use \"csi.storage.k8s.io/fstype\" instead"
E0201 06:29:50.203728       1 driver.go:122] "GRPC error" err="rpc error: code = AlreadyExists desc = Could not create volume \"pvc-89862b9e-f96a-40c0-9176-9986c7d37b7a\": Parameters on this idempotent request are inconsistent with parameters used in previous request(s)"
I0201 06:29:54.205298       1 controller.go:146] "\"fstype\" is deprecated, please use \"csi.storage.k8s.io/fstype\" instead"
E0201 06:29:54.403250       1 driver.go:122] "GRPC error" err="rpc error: code = AlreadyExists desc = Could not create volume \"pvc-89862b9e-f96a-40c0-9176-9986c7d37b7a\": Parameters on this idempotent request are inconsistent with parameters used in previous request(s)"

Environment

• Kubernetes version (use kubectl version): 1.24
• Driver version: v1.21.0-eksbuild.1

[ananthbh-te]

I verified the EBS CSI driver policy and it seems fine - #1140 (comment)

[ConnorJC3]

Hi @ananthbh-te , what you're seeing is likely an asynchronous creation failure due to KMS encryption of a volume failing.

If you've confirmed your policy is correct, I would also confirm that the ARN you are passing in your storage class points to a valid and existing KMS key in the same region as your volume.

If that doesn't help, could you provide the StorageClass parameters that you are using to create the volume?

[rifelpet]

FWIW Kops' prow k/k e2e testing has seen this same flaky error (see the linked issue above)

https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kops/16341/pull-kops-e2e-k8s-aws-calico/1756400222856024064

Feb 10 20:13:18.720: INFO: At 2024-02-10 20:08:18 +0000 UTC - event for aws46qr4: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
Feb 10 20:13:18.720: INFO: At 2024-02-10 20:08:18 +0000 UTC - event for aws46qr4: {ebs.csi.aws.com_ebs-csi-controller-84d99d85bf-gds8t_2697faee-bf5c-4097-9f9f-d34526ec0feb } Provisioning: External provisioner is provisioning volume for claim "volume-expand-6492/aws46qr4"
Feb 10 20:13:18.720: INFO: At 2024-02-10 20:08:18 +0000 UTC - event for aws46qr4: {persistentvolume-controller } ExternalProvisioning: Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
Feb 10 20:13:18.720: INFO: At 2024-02-10 20:08:22 +0000 UTC - event for aws46qr4: {ebs.csi.aws.com_ebs-csi-controller-84d99d85bf-gds8t_2697faee-bf5c-4097-9f9f-d34526ec0feb } ProvisioningFailed: failed to provision volume with StorageClass "volume-expand-6492cxrrm": rpc error: code = Internal desc = Could not create volume "pvc-8d24e563-c64c-45df-9f31-e7c3c9b8a994": failed to get an available volume in EC2: InvalidVolume.NotFound: The volume 'vol-029d5b5c03cd79b35' does not exist.
  	status code: 400, request id: 15206e5b-b1fd-4254-8dea-1f8be92d4cb8
Feb 10 20:13:18.720: INFO: At 2024-02-10 20:08:24 +0000 UTC - event for aws46qr4: {ebs.csi.aws.com_ebs-csi-controller-84d99d85bf-gds8t_2697faee-bf5c-4097-9f9f-d34526ec0feb } ProvisioningFailed: failed to provision volume with StorageClass "volume-expand-6492cxrrm": rpc error: code = AlreadyExists desc = Could not create volume "pvc-8d24e563-c64c-45df-9f31-e7c3c9b8a994": Parameters on this idempotent request are inconsistent with parameters used in previous request(s)

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.