[Fallback Support]Notation uses OCI image manifest to store the signature in the repository.

[yizha1]

What is the areas you would like to add the new feature to?

Notation CLI

Is your feature request related to a problem?

OCI artifact manifest was introduced in OCI image spec 1.1.0. It takes time for registries to be compliant to the 1.1.0 spec. There could be cases that registries decide not uplifted to 1.1.0 spec. For these registries not compliant to 1.1.0 spec, it will fail if notation CLI uses OCI artifact manifest to push the signatures.

What solution do you propose?

This feature is to request notation to support using OCI image manifest to store the signature in the registry that doesn't support OCI artifact manifest.

• Update signature spec to support additional format OCI image manifest
• Define and align the process that under which condition notation uses OCI image manifest
• Define the align the content for config property required by OCI image manifest

What alternatives have you considered?

No other alternative.

Notation can claim that only OCI artifact manifest is supported to store the signatures, but this will cause notation not broadly adopted.

Any additional context?

No response

[yizha1]

Illustration:
@toddysm @priteshbandi Based on our last discussion, does the following picture help to explain the situation?

[priteshbandi]

This fallback is not defined in OCI spec. The only fallback defined in OCI spec is based on referrers API. Do we want to devise fallback based on failure to push artifact manifest? If yes, why?

[yizha1]

This fallback is not defined in OCI spec. The only fallback defined in OCI spec is based on referrers API. Do we want to devise fallback based on failure to push artifact manifest? If yes, why?

You are right. This fallback is not defined in OCI spec. OCI artifact manifest is introduced in OCI image spec v1.1. Not all the registries are OCI image spec v1.1 compatible. If notation pushes signatures using OCI artifact manifest, the push fails. So the question is really about "Does Notation support sign container images/artifacts in registries not OCI image v1.1 compatible?".

[SteveLasker]

I'd suggest we should be checking for support before attempting to push either manifest. Having to create the manifest and depend on PUT errors is a lot of code to write and make calls just to find out it won't work.
Rather than create a manifest and PUT to determine support, why not check if it's supported first?

OCI Artifact manifest fallback is defined here: Backwards Compatibility
However, it's not clearly defined the referrers API is required to be implemented when a registry supports OCI Artifact Manifest
ORAS Artifact manifest described it as coupled, where the artifact manifest and referrers API were both implemented: API Discovery

However, @shizhMSFT pointed out that the /referrers API won't yet exist if the user is pushing to a new repository.

Question: what if we checked the root of the registry for the existence of the /referrers API? For registries like ACR and ECR that have a unique domain names for each registry, it would be the root of the domain. myregistry.azurecr.io/v2/referrers/<digest> while ghcr and docker hub would use the org namespace: docker.io/v2/myregistry/referrers/<digest>

[yizha1]

Add the link for related discussion at OCI: opencontainers/distribution-spec#365

[AaronFriel]

While I was investigating unrelated issues in using OCI Artifacts and the oras-project, I opened:

• Missing backwards compatibility and upgrade procedures for Artifact Manifest opencontainers/distribution-spec#368

[nima]

Notation fallback mechanism SFD on HackMD

Note that this is not going to work on ECR until it allows unknown fields (subject specifically for fallback) to pass through. In image-spec/considerations, it's suggested that:

Implementations that are reading/processing manifests or image indexes MUST NOT generate an error if they encounter an unknown property. Instead they MUST ignore unknown properties.

But I'm not sure if (being under considerations) this is part of the spec?

In any case, I'm looking at what changes need to be made in ECR in advance of 1/30/2023 that would allow this field, and thus make possible for notary to utilize the fallback mechanism if the move to oci-v1.1 is delayed.

[shizhMSFT]

Notation fallback mechanism SFD on HackMD

I've briefly reviewed the HackMD doc. The described behavior is similar to ORAS except the tag scheme (the fallback method of the Referrers API).

Major differences: oras-go always attempts to put the new referrers index first and then delete the old one while the HackMD doc suggests deleting the old index first and then put the new one. The oras-go implementation is more fault tolerant as it won't cause data / metadata loss if PUT the new index fails due to network issues or client / server errors. If we delete the old index first and then put the new one, we may lose all metadata if the final PUT fails.

[yizha1]

Notation fallback mechanism SFD on HackMD

@nima Question on the flow of "Attempt artifact manifest upload via oci-v1.1 spec", why do we need to check and upload oras config blob?

[nima]

Hi @yizha1, sorry for the delay, in response to your question in #444:

@nima Question on the flow of “Attempt artifact manifest upload via oci-v1.1 spec”, why do we need to check and upload oras config blob?

My understanding is that when we revert to a fallback (to use an image manifest, instead of an artifact manifest), then we need to construct an image manifest in its stead. And since an image spec mandates the manifest contain a config entry, we need to supply it one — even though we have no intention of ever using it. At that point, we need for it to reference some arbitrary config blob, and that config blob appears to be {} whose digest is sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a.

It's quite possible that I've missed something—in either your question or in my understanding in general—because I think you know all that I've explained better than I do. Let me know what I may have missed here, and I'll update the SFD.

[nima]

Hi @shizhMSFT, thank you for the review; that is indeed very important, I will update the SFD to reflect this for completeness.

[yizha1]

Hi @yizha1, sorry for the delay, in response to your question in #444:

@nima Question on the flow of “Attempt artifact manifest upload via oci-v1.1 spec”, why do we need to check and upload oras config blob?

My understanding is that when we revert to a fallback (to use an image manifest, instead of an artifact manifest), then we need to construct an image manifest in its stead. And since an image spec mandates the manifest contain a config entry, we need to supply it one — even though we have no intention of ever using it. At that point, we need for it to reference some arbitrary config blob, and that config blob appears to be {} whose digest is sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a.

It's quite possible that I've missed something—in either your question or in my understanding in general—because I think you know all that I've explained better than I do. Let me know what I may have missed here, and I'll update the SFD.

Thanks Nima. Your understanding is correct. The spec PR also described this notaryproject/specifications#217. My question is about the wording oras config blob. The configure descriptor is defined in OCI image spec. It is more accurate to call it OCI image configure blob, oras just provides an implementation for the spec. It's a minor comment.

[nima]

Oh! I'm so sorry I misunderstood that—I will update that immediately, and thank you for pointing it out.

[yizha1]

Closed as released in rc.2