Enable Kubernetes native support for AWS ECR container registries when using kube-aws

[whereisaaron]

TL;DR ECR cross-domain support is coming when k8s 1.3.0 releases and whichever version of kube-aws supports 1.3.0, provided by kube-aws patch #384 and k8s patch #24369. In the meantime you can use imagePullSecrets with a scheduled script like this one:
https://github.com/whereisaaron/kubernetes-aws-scripts/blob/master/create-ecr-imagepullsecret.sh

Kubernetes provides native support for AWS ECR container registries, including cross-region ECR support. The particular ECR features are that kubelets automatically obtains and regularly updates ECR login credentials (AWS ECR login credentials only last 12 hours), and uses those credentials when you specify an ECR repository for your image. Relevant Kurbernetes references below.

http://kubernetes.io/docs/user-guide/images/#using-aws-ec2-container-registry
kubernetes/kubernetes#19447
kubernetes/kubernetes#23298

However, this excellent feature is only activated if the kublet is started with the '--cloud-provider=aws' option and it appears the controller nodes generated by kube-aws do not have that option enabled (using kube-aws 0.7.0), CORRECTION: worker nodes do have this option.

/etc/systemd/system/kubelet.service

[Service]
Environment=KUBELET_VERSION=v1.2.3_coreos.0
Environment=KUBELET_ACI=quay.io/coreos/hyperkube
Environment="RKT_OPTS=--volume dns,kind=host,source=/etc/resolv.conf --mount volume=dns,target=/etc/resolv.conf"
ExecStart=/usr/lib/coreos/kubelet-wrapper \
--api-servers=http://localhost:8080 \
--network-plugin-dir=/etc/kubernetes/cni/net.d \
--network-plugin= \
--register-schedulable=false \
--allow-privileged=true \
--config=/etc/kubernetes/manifests \
--cluster_dns=10.3.0.10 \
--cluster_domain=cluster.local
Restart=always
RestartSec=10

The second requirement for this feature to work is the node AWS IAM roles also need ECR ReadOnly permissions added. The IAM 'AmazonEC2ContainerRegistryReadOnly' managed policy is exactly what is needed. Or the following actions could be added to the kube-aws generated role definition.

arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly

{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Action": [
            "ecr:GetAuthorizationToken",
            "ecr:BatchCheckLayerAvailability",
            "ecr:GetDownloadUrlForLayer",
            "ecr:GetRepositoryPolicy",
            "ecr:DescribeRepositories",
            "ecr:ListImages",
            "ecr:BatchGetImage"
        ],
        "Resource": "*"
    }]
}

[colhom]

@whereisaaron we have recently merged a PR (#384) which adds the ECR actions to the worker's IAM role. Kubernetes ECR integration should currently be working on master, though we have not cut it into a release yet.

[whereisaaron]

Good news. I also see from this patch that obtaining ECR credentials no longer relies on '--cloud-provider=aws' as the region info is taken from the ECR pull URL rather than the AWS instance info at http://169.254.169.254
kubernetes/kubernetes#24369

[whereisaaron]

It appears the vital patch #24369 to enable cross-region ECR is committed to the k8s master branch but not yet in a release version.

Using v1.2.3 on worker nodes the '--cloud-provider=aws' option is specified (why not the controller too?) and so the 'aws-ecr-key' credential provider plug-in is loaded. However the plugin appears to be the old version that works for the local region only. There is an error showing it is accessing the local region (ap-southeast-2) rather than the region of the pull URL (us-west-2).

3b, skipping: failed to "StartContainer" for "busybox" with ErrImagePull: "API error (500): Get https://555123456789.dkr.ecr.us-west-2.amazonaws.com/v2/busybox/manifests/latest: no basic auth credentials\n"

Jun 03 20:22:36 ip-172-28-8-232.ap-southeast-2.compute.internal kubelet-wrapper[1529]: I0603 20:22:36.636875    1529 plugins.go:56] Registering credential provider: aws-ecr-key
Jun 03 20:22:41 ip-172-28-8-232.ap-southeast-2.compute.internal kubelet-wrapper[1529]: I0603 20:22:41.649537    1529 provider.go:91] Refreshing cache for provider: *aws_credentials.ecrProvider
Jun 03 20:22:42 ip-172-28-8-232.ap-southeast-2.compute.internal kubelet-wrapper[1529]: caused by: Post https://ecr.ap-southeast-2.amazonaws.com/: dial tcp: lookup ecr.ap-southeast-2.amazonaws.com: no such host

cat /etc/systemd/system/kubelet.service

[Unit]
Requires=docker.service
After=docker.service

[Service]
Environment=KUBELET_VERSION=v1.2.3_coreos.0
Environment=KUBELET_ACI=quay.io/coreos/hyperkube
Environment="RKT_OPTS=--volume dns,kind=host,source=/etc/resolv.conf --mount volume=dns,target=/etc/resolv.conf"
ExecStart=/usr/lib/coreos/kubelet-wrapper \
--api-servers=https://172.28.0.50:443 \
--network-plugin-dir=/etc/kubernetes/cni/net.d \
--network-plugin= \
--register-node=true \
--allow-privileged=true \
--config=/etc/kubernetes/manifests \
--cluster_dns=10.3.0.10 \
--cluster_domain=cluster.local \
--cloud-provider=aws \
--kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml \
--tls-cert-file=/etc/kubernetes/ssl/worker.pem \
--tls-private-key-file=/etc/kubernetes/ssl/worker-key.pem
Restart=always
RestartSec=10

[whereisaaron]

The AWS cross-region ECR support is included in 1.3.0, currently at alpha 5.
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md/#v130-alpha4

[whereisaaron]

While we are waiting for 1.3.0 to hit kube-aws here is a script that you can schedule that will create an K8s registry secret that you can use with imagePullSecret in pod definitions.

https://github.com/whereisaaron/kubernetes-aws-scripts/blob/master/create-ecr-imagepullsecret.sh

[aaronlevy]

Is there any work to be done here, or is this just waiting on v1.3?

[colhom]

@aaronlevy yep, we're waiting on v1.3 before we expose cluster.yaml directives related to ECR pull secrets.

[FrederikNJS]

Kubernetes is up to v1.3.5 now, has there been any progress on this?

[Thermus]

This is required to run daemonsets with images from the ECR. Fixing this is very easy when done upstream. Please do so. I just had to do this. And I don't think you don't need to expose anything. Just add the required policies to the IAM role's inline policy and add that flag to the kubelet systemd unit.

[vinayk06]

can you please explain what flag should be added to the kubelet systemd. sample would be helpfull

[Thermus]

The kubelet systemd service file just needs --cloud-provider=aws \ to be added (remote the , if it goes onto the end of the argument list for you. I put the whole ling in between the existing arguments, into the same place as in the unit file on the worker nodes.)

The IAM policy looks like that:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetRepositoryPolicy",
                "ecr:DescribeRepositories",
                "ecr:ListImages",
                "ecr:BatchGetImage"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

[aaronlevy]

I believe this issue has been resolved as noted in #620 -- I'm going to close the issue as resolved, but please let me know if it needs to be re-opened

[mabushey]

This is still broke, why was is closed out w/o being fixed?

[colhom]

kube-aws development has moved to https://github.com/kubernetes-incubator/kube-aws