IAM role assumption doesn't seem to work in v0.5.1

[jmeickle]

Role assumption was addressed by this PR: https://github.com/kubernetes-incubator/external-dns/pull/524/files

Which is stated as released as of v0.5.0 in the release notes. However, it isn't working for me in v0.5.1 with the suggested configuration. I get these logs out of the pod:

time="2018-05-30T20:15:04Z" level=info msg="Connected to cluster at $URL"
time="2018-05-30T20:15:04Z" level=info msg="Assuming role: arn:aws:iam::XXX:role/qbernetes-staging-external-dns"
time="2018-05-30T20:15:04Z" level=error msg="NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"

This is the relevant part of the Deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: external-dns
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: external-dns
      annotations:
         # NOTE: This is syntax for kiam
        iam.amazonaws.com/role: qbernetes-staging-external-dns
    spec:
      serviceAccountName: external-dns
      containers:
      - name: external-dns
        image: registry.opensource.zalan.do/teapot/external-dns:v0.5.1
        args:
        - --source=service
        - --provider=aws
        - --policy=upsert-only
        - --aws-zone-type=public
        - --registry=txt
        - --txt-owner-id=qbernetes-staging
        - --aws-assume-role=arn:aws:iam::XXX:role/qbernetes-staging-external-dns
        - --log-level=debug

To check that kiam itself is working, I hop inside the pod with:

$ kubectl exec -it external-dns-6f9dc578c6-pg2h2 -- /bin/sh

I confirm that I have access to assume the role with:

$ wget -qO - http://169.254.169.254//latest/meta-data/iam/security-credentials/
qbernetes-staging-external-dns/ # 

To confirm that role works as expected, I install the AWS CLI inside the container:

$ apk -Uuv add groff less python py-pip && pip install awscli

Then I run this:

$ aws --debug route53 list-hosted-zones

[... lines omitted ...]
2018-05-30 20:23:16,463 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: iam-role
2018-05-30 20:23:16,466 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTP connection (1): 169.254.169.254
2018-05-30 20:23:16,468 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET /latest/meta-data/iam/security-credentials/ HTTP/1.1" 200 30
2018-05-30 20:23:16,469 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTP connection (1): 169.254.169.254
2018-05-30 20:23:16,472 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET /latest/meta-data/iam/security-credentials/qbernetes-staging-external-dns HTTP/1.1" 200 591

I can see that boto is using the role, and the commands are working. Yet the external DNS pod is still reporting NoCredentialProviders with this role, even when it's set to * like so:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "debug",
            "Effect": "Allow",
            "Action": [
                "route53:*"
            ],
            "Resource": "*"
        }
    ]
}

Curious if there is something obvious I've missed, or if there is some bug in how external-dns is trying to assume/use roles.

[hjacobs]

I'm confused by your roles: both the node (kiam) and the option to External DNS have the same role name "qbernetes-staging-external-dns". Having the iam.amazonaws.com/role annotation and the --aws-assume-role option at the same time only makes sense if they refer to different roles. Also the node role (kiam) needs to have sts:AssumeRole permissions in order for --aws-assume-role to work.

Maybe you don't even need --aws-assume-role? Check our setup for reference: https://github.com/zalando-incubator/kubernetes-on-aws/blob/dev/cluster/manifests/external-dns/deployment.yaml

[jmeickle]

I had added --aws-assume-role because it wasn't working without it, so I assumed that argument existed because explicit role assumption was expected. I get the same behavior without using that argument, except without the log messaging about assuming role. For instance, after updating the Deployment to not have that argument, this is what I get on the AWS CLI:

/ # aws sts assume-role --role-arn arn:aws:iam::XXX:role/qbernetes-staging-external-dns --role-session-name DEBUG

An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::XXX:assumed-role/qbernetes-staging-external-dns/kiam-kiam is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::XXX:role/qbernetes-staging-external-dns

So kiam is definitely working for kubectl exec inside the pod, but it's either not working for the original process in the pod, or not working specifically for external-dns.

[jmeickle]

Oh wow, I think I found the issue:

aws cli debug output:

2018-05-31 16:09:32,669 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTP connection (1): 169.254.169.254
2018-05-31 16:09:32,671 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET /latest/meta-data/iam/security-credentials/ HTTP/1.1" 200 30

after adding debug logging and cert chain logging to external-dns and rebuilding container:

2018/05/31 16:07:13 DEBUG: Request ec2metadata/GetMetadata Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /latest/meta-data/iam/security-credentials HTTP/1.1

On a kubectl exec:

/ # wget -qO - http://169.254.169.254/latest/meta-data/iam/security-credentials/
qbernetes-staging-external-dns/

/ # wget -qO - http://169.254.169.254/latest/meta-data/iam/security-credentials
Node-Qbernetes-Staging-NodeQbernetesStagingInstanc-GPT94VFBQBLK/

boto is calling the metadata API in some way that has a slash at the end, so it gets qbernetes-staging-external-dns and the correct permissions. external-dns AWS SDK go library is calling without the slash and seeing the instance role.

I think this is ultimately a kiam issue, in that case...

[hjacobs]

@Eronarn maybe it's the same (or related) as jtblin/kube2iam#130 ?

[jmeickle]

Yep, looks like there is a corresponding issue here that I didn't see because it's been closed: uswitch/kiam#43

Thanks for your patience! Closing.