Cannot get efs-csi-node to Assume Role

[sinkr]

/kind bug

What happened?

Hello, thanks for your help in advance!

Chart 2.2.7 (I have also tried chart 2.1.5 to no avail):

efs-csi-controller, according to CloudWatch, correctly assumes the EFS role using the prescribed policy in the documentation here and create an access point, however, efs-csi-node with the same annotation will not assume the rule.

When I exec into an efs-csi-node pod, install awscli and perform aws sts get-caller-identity, the assumed role is correct.

The annotation is set correctly on the DaemonSet's annotation, and the AWS_ROLE_ARN environment variable are correctly set, however, while I see efs-csi-controller correctly assuming the role for activities such as CreateAccessPoint, anything coming from efs-csi-node comes across as ANONYMOUS_PRINCIPAL.

What you expected to happen?
I expect efs-csi-node to assume the EFS-specific role using the prescribed policy in the documentation just like efs-csi-controller correctly does, per the annotation in the SA + what is shown on the pod under AWS_ROLE_ARN.

How to reproduce it (as minimally and precisely as possible)?

1. Helm install chart v2.2.7
2. Create the IAM role w/ the prescribed policy in the documentation here.
3. Update the trust policy to allow both node and controller service accounts to assume the IAM role.
4. Apply the EFS policy allowing the IAM role access to the EFS filesystem.
5. Annotate the service accounts for both the controller and node w/ the IAM role from step 2 above.
6. Create the storage class using the FSid in examples/kubernetes/dynamic_provisioning/specs.
7. Create the pod using pod.yaml in examples/kubernetes/dynamic_provisioning/specs
8. Inspect the efs-csi logs from Kubernetes (permission denied)
9. Inspect the CloudWatch logs for all elasticfilesystem activity in the last minute (see that efs-csi-controller correctly assumes the EFS role and that efs-csi-node comes across as ANONYMOUS_PRINCIPAL.

Anything else we need to know?:

efs-csi-controller correctly assumes the role, but
efs-csi-node does not attempt to assume the role, issues no error, even at logLevel 9, instead attempting to mount EFS via ANONYMOUS_PRINCIPAL.

Environment

• Kubernetes version (use kubectl version):
  EKS 1.22

• Driver version:
  1.4.0

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sinkr]

Ping, this is still an issue with 1.5.4.

[sinkr]

/reopen

[k8s-ci-robot]

@sinkr: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[nicoodle]

Seeing the same on v1.5.6

[gadiener]

Same on v1.5.7

[andrewhharmon]

as a workaround i think you can specify the iam mount_option flag on the storageClass or the PV. That seemed to resolve it for me. I set a file system policy and it stated failing bc it was coming in as anonymous.

#280

[goyertp]

Hello, I have added the iam option. Creating the pvc and mounting the volume in my pod is successful. But a closer look at aws organisation trail reveals the problems.

The access is anonymous.

Here is an example:
{type=AWSAccount, principalid=, arn=null, accountid=ANONYMOUS_PRINCIPAL, invokedby=null, accesskeyid=null, username=null, sessioncontext=null}

My StorageClass seems to be set up correctly:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs
provisioner: efs.csi.aws.com
mountOptions:
  - tls
  - iam

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.