Optional support for trailing slash on AWS metadata service

[burdara]

Some instances have experienced difference response behavior when calling the AWS metadata without a trailing slash. I have included an example of this behavior I observed below. AWS was not helpful when I opened a ticket asking them to look into the metadata service behavior.

Trailing slash example

Instance A (AMI k8s-1.7-debian-jessie-amd64-hvm-ebs-2017-07-28, kops cluster, m4.large)

# On this instance, I see the expected behavior:
#  CALL 1 results in listed the available IAM role attached to the instance.  I.e. Expected.
#  CALL 2 results in a 301 redirect.  I.e. Expected.

# CALL 1
root@ip-foo:/home/admin# curl -is http://169.254.169.254/latest/meta-data/iam/security-credentials/
HTTP/1.0 200 OK
Content-Type: text/plain
Accept-Ranges: bytes
ETag: "3138592401"
Last-Modified: Sun, 07 Jan 2018 15:51:16 GMT
Content-Length: 40
Connection: close
Date: Sun, 07 Jan 2018 16:50:54 GMT
Server: EC2ws

nodes.kops18-test.us-west-2.example.com

# CALL 2
root@ip-foo:/home/admin# curl -is http://169.254.169.254/latest/meta-data/iam/security-credentials
HTTP/1.0 301 Moved Permanently
Location: http://169.254.169.254/latest/meta-data/iam/security-credentials/
Content-Length: 0
Connection: close
Date: Sun, 07 Jan 2018 16:51:17 GMT
Server: EC2ws

--no output--

Instance B (An encrypted copy of AMI k8s-1.8-debian-stretch-amd64-hvm-ebs-2017-12-02, kops cluster, m5.xlarge)

# On this instance, I see different routing behavior when using the same calls as above.
#  CALL 1 Returns the listed of available IAM role attached to the instance. I.e. Expected.
#  CALL 2 WITHOUT A REDIRECT, RETURNS THE AVAILABLE ROLE ATTACHED TO INSTANCE. I.e. NOT expected behavior.

root@ip-bar:/home/admin# curl -is http://169.254.169.254/latest/meta-data/iam/security-credentials/
HTTP/1.1 200 OK
Content-Type: text/plain
Accept-Ranges: none
Last-Modified: Sun, 07 Jan 2018 16:46:13 GMT
Content-Length: 29
Date: Sun, 07 Jan 2018 16:51:05 GMT
Server: EC2ws
Connection: close

nodes.kops18-test.us-west-2.example.com

root@ip-bar:/home/admin# curl -is http://169.254.169.254/latest/meta-data/iam/security-credentials
HTTP/1.1 200 OK
Content-Type: text/plain
Accept-Ranges: none
Last-Modified: Sun, 07 Jan 2018 16:46:13 GMT
Content-Length: 29
Date: Sun, 07 Jan 2018 16:51:33 GMT
Server: EC2ws
Connection: close

nodes.kops18-test.us-west-2.example.com

Additional background.

I noticed this behavior attempting to used kube2iam with cluster-autoscaler on the kops k8s 1.8 cluster with Instance B.

From a test cluster-autoscaler container with annotated role kops-cas-test-role for kube2iam.

# cleaned up output to make it more readable.
root@test-cas:/# curl http://169.254.168.254/latest/meta-data/iam/security-credentials/
kops-cas-test-role

# the above call returns proper role by hitting `https://github.com/jtblin/kube2iam/blob/master/server/server.go#L290`

root@test-cas:/# curl http://169.254.168.254/latest/meta-data/iam/security-credentials
nodes.kops18-test.us-west-2.example.com

# the above call hits `https://github.com/jtblin/kube2iam/blob/master/server/server.go#L293`
# Rather than doing a redirect to `iam/security-credentials/`, due to metadata service behavior above, it returns value form the base kops role.

kube2iam log output from cluster-autoscaler

time="2018-01-03T18:18:27Z" level=debug msg="Proxy ec2 metadata request" metadata.url=169.254.169.254 req.method=GET req.path=/latest/meta-data/iam/security-credentials req.remote=100.110.128.2
time="2018-01-03T18:18:27Z" level=info msg="GET /latest/meta-data/iam/security-credentials (200) took 867314 ns" req.method=GET req.path=/latest/meta-data/iam/security-credentials req.remote=100.110.128.2 res.duration=867314 res.status=200
time="2018-01-03T18:18:27Z" level=error msg="Invalid role: does not match annotated role" ns.name=kube-system params.iam.role=nodes.kops18-test.us-west-2.example.com pod.iam.role="arn:aws:iam::000000000:role/kops-cas-test-role" req.method=GET req.path=/latest/meta-data/iam/security-credentials/nodes.kops18-test.us-west-2.example.com req.remote=100.110.128.2
time="2018-01-03T18:18:27Z" level=info msg="GET /latest/meta-data/iam/security-credentials/nodes.kops18-test.us-west-2.example.com (403) took 266967 ns" req.method=GET req.path=/latest/meta-data/iam/security-credentials/nodes.kops18-test.us-west-2.example.com req.remote=100.110.128.2 res.duration=266967 res.status=403

Fix summary

I have added a optional, unmatched forward slash regex to the end of the meta-data/iam/security-credentials proxy route.

[burdara]

Will review the test failures shortly and resolve.

[coveralls]

Coverage remained the same at 18.482% when pulling 7e446ed on burdara:metadata-trailing-slash into 7ebf457 on jtblin:master.

[coveralls]

Coverage remained the same at 18.482% when pulling 8656950 on burdara:metadata-trailing-slash into 7ebf457 on jtblin:master.

[bhvr-fmarceau]

I'm having the same issue with m5 instnaces, I was able to reproduce it!

+1 for this issue!

[Folkoff-Martin-bah]

replicated as well

+1

[chrismoos]

Seeing this also on m5 instances.

[michaelajr]

Had the exact same issue with our encrypted copy of the Debian image on M5s. Reverting to m4s fixed the issue.

[pms1969]

likewise: m4's work, m5's don't. any update on when this might be accepted and released?

[faraazkhan]

This is currently stopping us from upgrading to M5s. We are running our own container with this patch in non prod now, but would like to use an official image for production. So just another +1

[nhoughto]

Been 2 months today since this was opened =(
+1 we would like this too

[burdara]

In the meantime, I have pushed this fix to my docker hub burdara/kube2iam:0.9.1. Feel free to use or just build your own. 😄

/cc @jtblin can you please look at this PR #119 (or #121) and merge the fix you prefer?

[jtblin]

Sorry for the lag, I've merged #130 which does the same and will deploy a new version.