Image Signing Support in ECR

[DrFaust92]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
Support for storing image signatures in ECR.

Which service(s) is this request for?
Storing container image signatures in ECR, verification of signatures in ECS/EKS.

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Notary V1 is the currently available community-supported tool that would allow for signing and verifying OCI/Docker images, but it requires standing up a separate service and a lot of heavy lifting that each customer must do to setup and maintain.

Are you currently working around this issue?
N/A

Additional context

Update from ECR team (10/14):

We are actively working on adding support for container image signing in ECR. To deliver this feature, AWS is participating in two parallel open source efforts to deliver support for storing signatures (and other related artifacts) in an OCI registry and performing signature validation in a container orchestrator so we can launch a solution that will be compatible across container orchestrators and OCI registries.

• To enable the storage and discovery of “reference artifacts”, such as signatures, in an OCI registry, we are working with the ORAS project to define a new specification for OCI Artifacts. Last month, we were excited to announce an initial draft release for that project!
• We are also participating in the design and development of Notary V2, to define industry standards for signing and validating images that can be implemented in the tools used to build images today and container orchestrators like EKS & ECS.

We will update this issue when we reach key milestones, but for an up to date picture of our progress, please take a look at the respective projects. We’re always looking for feedback and collaborators, so join us in the oras-project/artifacts-spec & notaryproject/notaryproject repositories!

[jtoberon]

Thanks for feedback, @DrFaust92. We've started to discuss how we want this to work for our customers. It's a surprisingly complicated topic though, so we don't have a proposal to share yet. We're going to leave this open as a placeholder.

[DrFaust92]

Any update on this?

[omerfsen]

Would be great to see it on AWS ECR. Also I think until it is out we can run our own notary server and then after signing docker image via Notary then push it to ECR

[jmb12686]

Any update or insight into the status of this for ECS? Image SHA tracking was announced for ECS https://aws.amazon.com/about-aws/whats-new/2019/10/amazon-ecs-now-supports-ecs-image-sha-tracking/ , however it's not clear if this fulfills the trusted content requirement.

[omieomye]

Update: as part of a broader community 'Notary v2' initiative, ECR will participate and contribute with a view to apply that specification to our effort tracked by this issue. Its an open group with multiple cloud and on-premise vendors working together, with the kickoff meeting held on 12/12 here in Seattle.

[jmb12686]

@omieomye , Thank you for providing an update and transparency into the current state of container signing within the broader community. Aside from listening to the kick-off meeting, how can users get involved in the discussion?

[jtoberon]

Call in details for the OCI weekly meeting is available here: https://github.com/opencontainers/org. You also can join the relevant IRC and Slack channels, which are linked from the same github page.

[jonassteinberg1]

Am I correct in thinking that notary cannot be used with ecr still?

[chrisdipesa]

Yup. https://awscloudcontainersconference.splashthat.com/ Everyone should attend this event.

Security Best Practices with Amazon ECR
Omar Paul, Sr Product Manager, ECR

We have questions for Omar!

[omieomye]

Our progress on Notary is tracked by this issue, and we're actively participating towards a Notary v2 specification. On the summit presentation, I would love to get feedback what the ECR community wants us to tackle. Tweet or DM @omieomye and we'll go from there.

[w8mej]

Currently slated 2021 with Notary v2 per Omar's presentation linked by @chrisdipesa above. Are there any other compensating controls one could perform to meet this need until 2021?

[jnaulty]

Hey @omieomye and @chrisdipesa
I'm curious to know if there are any slides or recording from the summit presentation. The links provided no longer work.

[kapilt]

seems this issue is missing any context on why v2, so adding in some links

high level blog post on v2 - https://www.docker.com/blog/community-collaboration-on-notary-v2/
v2 requirements - https://github.com/notaryproject/requirements
working group meeting notes - https://hackmd.io/_vrqBGAOSUC_VWvFzWruZw

[renkenk]

+1

[knksmith57]

With the release of ECR Public, this seems more relevant and valuable than ever.

[bjethwan]

Any update on this?
Is there a tentative release date attached to this?
It's more 5 months since the last comment on this issue.

[dlorenc]

For people here interested in container signing in general, feel free to checkout github.com/sigstore/cosign. It should already work well with ECR.

[ari-becker]

For people here interested in container signing in general, feel free to checkout github.com/sigstore/cosign. It should already work well with ECR.

@dlorenc does it work with ECR? The project doesn't specify ECR as being supported.

[06kellyjac]

$ cosign sign -key cosign.key dlorenc/demo
Enter password for private key:
Pushing signature to: index.docker.io/dlorenc/demo:sha256-87ef60f558bad79beea6425a3b28989f01dd417164150ab3baab98dcbf04def8.cosign

It looks like cosign pushes an "image" holding the signing details to a label called sha256-blahblah.cosign so it's only using very normal features of a basic image registry

cosign uses go-containerregistry for registry interactions, which has excellent support, but some registries may have quirks.

As long as ECR doesn't behave wildly different to all the other registries it should work just fine.
I saw one ECR related issue in https://github.com/google/go-containerregistry but that's been resolved

[dlorenc]

I don't have any AWS credentials so I haven't been able to test it. That "supported" list is really just a list of registries people have confirmed it works with. If anyone tries it out, please feel free to send a PR adding the entry!

We don't do anything not allowed by the existing OCI specifications so I'd be surprised if it doesn't work, but a few registries do have some quirks here.

[coultn]

Hi everyone, GM of ECR here. We are still working on this. We don't have more details we can share beyond what @michaelb990 already said, but please do feel free to follow along with what's happening at oras-project/artifacts-spec and notaryproject/notaryproject.

[jlbutler]

Quick update on this, sorry it could have been made sooner. Since the last update, members of the ECR team joined others in the formation of the OCI's working group for reference types. This resulted in new 1.1 versions of the OCI Image and Distribution specs, providing OCI Artifacts and a new Referrers API. With these features, we can leverage an open specification for managing image signatures as well as other artifact types in ECR. We've partnered with another service team to bring a managed image signing experience to AWS, based upon Notary v2. Both of these projects are nearing a GA, at which point we can share with you all the work we've been doing. We will update here when we have more specifics to share. Thanks for your patience, and more soon!

[NikolaySokolov]

@jlbutler is there any updates on this topic? We are at the point either waiting for this to be available or creating our own solution. Can you please give us more details on when it can be expected?

[SamirPS]

@jlbutler @michaelb990 Are there any updates on the topic of the ORAS artefact? Especially for attaching a file to an ECR image with oras attach?

[jlbutler]

Hi all, sorry for the delay. Quite a bit has changed while we continue to work upstream on the OCI 1.1 specs. As things are still in flux there, we've been working with the client-side fallback references support for ECR. That's now working, which unblocks some pieces of the overall picture.

The Notary v2 client has now implemented support for references within the scope of OCI 1.0, and you can use the 1.0.0-rc2 release of Notation with ECR. We are still working on a more managed solution, but this is something to look at to get an idea of the workflow of signing an image. We'll put a blog out on that soon, and as soon as we have more to share about a managed solution, we'll certainly update here.

[jlbutler]

@jlbutler @michaelb990 Are there any updates on the topic of the ORAS artefact? Especially for attaching a file to an ECR image with oras attach?

The ORAS Artifact work moved into the OCI, but that said you can use the latest oras CLI to attach with ECR. Note until the 1.1 spec is released, you'll need to use the --image-spec v1.1-image option.

e.g.

$ oras attach --image-spec v1.1-image --export-manifest test-manifest.json --artifact-type test/example 1234.dkr.ecr.us-west-2.amazonaws.com/oras-test:latest ./test.txt
Uploading 524b162ec8ad test.txt
Uploaded  524b162ec8ad test.txt
Attached to [registry] 1234.dkr.ecr.us-west-2.amazonaws.com/oras-test@sha256:4c524d1407ad83d60e4324050668d3ee77af96f2a88d0ff418ecbbcad502449d
Digest: sha256:d9cc47b48144f19dd56ecd757286208ac2f0dc33e22dbe50cd41055214dfab25

[SamirPS]

@jlbutler i have this error with ecr public

oras attach -v --artifact-type "application/spdx+json" public.ecr.aws/rocksdev/artifact-test:lunar empthyone.json -u AWS -p $(aws ecr-public get-login-password --profile ociimage) --image-spec v1.1-image --export-manifest test-manifest.json
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Preparing empthyone.json
Exists    44136fa355b3 application/spdx+json
Exists    ca3d163bab05 empthyone.json
Exists    678c136e4e91 application/vnd.oci.image.index.v1+json
Uploading a8ddde0d870c application/vnd.oci.image.manifest.v1+json
Error: PUT "https://public.ecr.aws/v2/rocksdev/artifact-test/manifests/sha256:a8ddde0d870c85f4afd502f3df4b9695a94b9e83c0810eb211db1c2e835b3ffa": response status code 405: unsupported: Invalid parameter at 'ImageManifest' failed to satisfy constraint: 'Invalid JSON syntax'

[jlbutler]

Hi @SamirPS. Sorry about that, I should have been more specific. In ECR, we have done some work to support the recent changes in ORAS client and client-side artifact support in general. This work is ongoing in ECR Public at the moment, and will be available soon. I will post back here when that work is complete. Thanks!

[jlbutler]

Hi @SamirPS. Sorry about that, I should have been more specific. In ECR, we have done some work to support the recent changes in ORAS client and client-side artifact support in general. This work is ongoing in ECR Public at the moment, and will be available soon. I will post back here when that work is complete. Thanks!

I completely neglected to reply back here as I said I would - apologies @SamirPS. Client-side reference types are supported in ECR Public now.

[SamirPS]

@jlbutler Thanks for the information

[jlbutler]

Hi all. It's been quite a long road, but we have launched a managed signing solution today with AWS Signer's new Container Image Signing capability.

Launch blog is a good place to get started, please let us know what you think. Thanks so much for your patience along the way!

https://aws.amazon.com/blogs/containers/announcing-container-image-signing-with-aws-signer-and-amazon-eks/