Mark feature "support OCI Referrer API" as experimental feature #659

yizha1 · 2023-05-09T06:00:27Z

What is the areas you experience the issue in?

Notation CLI

What is not working as expected?

Currently Referrer API is by default verified by Notation since v1.0.0-rc.4. For example, when using notation sign, if registries support referrer API, notation will not use an image index pushed to a tag described by the referrers tag schema. Since OCI spec 1.1 (including both image spec and distribution spec) is not officially released yet, support of Referrer API could lead to breaking changes or portability issue after Notation v1 is released. On the other hand, users may be confused that sometimes image index is pushed, sometimes not, which is an issue of consistency.

What did you expect to happen?

Mark feature "support OCI Referrer API" as experimental feature, and document it properly. Users can use experimental features only after env variable NOTATION_EXPERIMENTAL=1 is set.
The default behavior of Notation is always using an image index pushed to a tag described by the referrers tag schema. If users set env variable NOTATION_EXPERIMENTAL=1, Notation will check the support of OCI referrer API first, if registries support Referrer API, then notation will not push image index; if registries don't support Referrer API, notation will push image index as the default behavior.

How can we reproduce it?

N/A

Describe your environment

WSL

What is the version of your Notation CLI or Notation Library?

Notation v1.0.0-rc.4

The text was updated successfully, but these errors were encountered:

yizha1 · 2023-05-09T06:02:16Z

@toddysm @iamsamirzon @sajayantony @shizhMSFT @vaninrao10 This issue was created per discussion during community call on 5/8/2023. Please take a look, and this issue should be addressed in Notation v1 release

Two-Hearts · 2023-05-09T06:24:09Z

Thanks @yizha1, I will take this one.

Two-Hearts · 2023-05-09T08:04:36Z

I'm trying to summarize behaviors between Notation versions:
Sign as of Notation v1.0.0-rc.4 (current version):

If store signature with image manifest: Try Referrers API first, if not supported, automatically fallback to referrers tag schema.
If store signature with artifact manifest: ping Referrers API, if failed (i.e. Referrers API is not supported), then the whole Sign process is failed, no fallback would occur.

Verify as of Notation v1.0.0-rc.4 (current version):
Regardless of manifest type, always try Referrers API, if not supported, automatically fallback to referrers tag schema.

What we want in v1.0 for Sign:

If experimental flag is disabled (NOTATION_EXPERIMENTAL=0), always use referrers tag schema to store signatures.
If experimental flag is enabled (NOTATION_EXPERIMENTAL=1), then try referrers API first, if not supported, automatically fallback to referrers tag schema.

What we want in v1.0 for Verify:
Q: what is the expected behavior here? @yizha1 /cc: @iamsamirzon @vaninrao10 @sajayantony @toddysm @shizhMSFT @FeynmanZhou

yizha1 · 2023-05-09T08:48:32Z

For notation verify in v1.0, the behavior is similar to notation sign. The default is using image index, if experimental flag is enabled (NOTATION_EXPERIMENTAL=1), then try referrers API first, if not supported, automatically fallback to use the image index. This applies to all the commands including notation sign, notation verify, notation ls, and notation inspect

Two-Hearts · 2023-05-09T09:09:21Z

For notation verify in v1.0, the behavior is similar to notation sign. The default is using image index, if experimental flag is enabled (NOTATION_EXPERIMENTAL=1), then try referrers API first, if not supported, automatically fallback to use the image index. This applies to all the commands including notation sign, notation verify, notation ls, and notation inspect

(Just an open discussion here.) I'm not sure if this is what we expect, but in the below scenario:
Suppose the registry supports the Referrers API, user sets NOTATION_EXPERIMENTAL=1, then sign their artifact. The Sign succeeded without pushing the image index as Referrers API is used. Then, the user turns off experimental by setting NOTATION_EXPERIMENTAL=0. Next, if the user types notation list it would show empty, as Notation is default to referrers tag schema now. If the user types notation verify, the verification would fail with error: no signature is associated.... Now, if the user turns on the experimental again, both notation list and notation verify would work as Referrers API is turned on again.

In the above case, it seems that the result of list and verify depends on the value of NOTATION_EXPERIMENTAL. IMO, this might not be a very good user experience? So I think, maybe we could just put Referrers API behind experimental in Sign. For other commands, we still try Referrers API first, if failed, automatically fallback to tag schema.

shizhMSFT · 2023-05-09T10:41:56Z

We can assume the use of NOTATION_EXPERIMENTAL is fairly consistent and most customer will not enable this flag.

iamsamirzon · 2023-05-09T16:53:40Z

2. rimental flag is enabled (NOTATION_EXPERIMENTAL=1), then try referrers API first, if not supported, automatically fallback to referrers tag schema.

I like the proposal from @Two-Hearts about having the Notation Verify check for referrers API first and if not available use the image-index method. As long as this does not cause inconsistent results for registries this may be a good path. Essentially be more flexible when verifying signatures so users do not have to change their deployment time checks (modify scripts or flags for verify) when their registries start supporting 1.1 based signatures

priteshbandi · 2023-05-09T23:00:18Z

My 2 cents, for v1.0

For Sign:

If experimental flag is disabled (NOTATION_EXPERIMENTAL=0), always use referrers tag schema to store signatures. Also, customers should not be able to use --signature-manifest flag
If experimental flag is enabled (NOTATION_EXPERIMENTAL=1) and customer uses --signature-manifest artifact then try referrers API first, if not supported fail the signing. The default behavior without --signature-manifest flag should be same as with experimental disabled.

For Verify, inspect, list aka any command that only read's content from registry:

Always try Referrers API, if not supported, automatically fallback to referrers tag schema. --signature-manifest flag is not supported

Rationale:

How signatures are stored should be a conscious call that user makes i.e. notation should not make any assumptions, which means for sign call we should take intent from users on what mechanism they prefer to store signature. This is required for cross tool compatibility.
For operations such as verify, inspect, we can do automatic fallback. This provides backward compatibility for users who moves from signing using image-manifest to use artifact manifest.

IMO we should fix this ASAP as its breaking customer experience. Should we have RC5 with this fix?

yizha1 · 2023-05-10T00:32:02Z

My 2 cents, for v1.0

For Sign:

If experimental flag is disabled (NOTATION_EXPERIMENTAL=0), always use referrers tag schema to store signatures. Also, customers should not be able to use --signature-manifest flag

The rationale for commands verify, ls and inspect makes sense. But Referrer API doesn't have to be with specific manifest type, that means, no matter what signature manifest is used. We have another issue to remove --signature-manifest flag, since artifact manifest has been removed by recent OCI image spec v1.0.0-rc.3 release.

priteshbandi · 2023-05-10T01:13:59Z

@yizha1 Yes, we should remove --signature-manifest flag and introduce a new experimental flag --use-referrers-api (name TBD) using which users can control if they want to push a signature using tag-schema or referrers API.

Rationale:

OCI spec removed artifact manifest.
For backward compatibility in CI/CD tooling. Think about a scenario where a user is using a managed 3rd party registry like ACR, ECR, docker hub and has CICD tooling around it, a notation to sign an image, and a signature verification tool(either homegrown or 3rd party) that supports only tag schema to verify the image.
As of today, the registry doesn't support referrers API so notation signs with tag schema and verification tool verify image using tag schema. One day, the registry added support for referrers API and suddenly notation started signing using referrers API, which the verification tool doesn't understand, and that broke the user's CICD.

Two-Hearts · 2023-05-10T02:07:49Z

Thanks @priteshbandi, to summarize what we want in RC.5:
For Sign:

--signautre-manifest flag would be removed, we will not support signing with OCI artifact manifest anymore.
Adding a new flag, --use-referrers-api (name TBD), behind experimental:
- When both NOTATION_EXPERIMENTAL and --use-referrers-api are set, Notation would try the Referrers API first, if failed, automatically fallback to referrers tag schema.
- Otherwise, Sign will be using referrers tag schema as default.

For other commands that consume signatures, notation list, notation verify, etc.:

Notation still supports OCI artifact manifest.
Always try the Referrers API first, if failed, automatically fallback to the referrers tag schema.

Does the above sound correct to you? (the new flag's name --use-referrers-api lgtm, but might need some further discussions.) /cc: @iamsamirzon @vaninrao10 @toddysm @sajayantony @shizhMSFT @yizha1 @FeynmanZhou

shizhMSFT · 2023-05-10T03:37:39Z

Always try the Referrers API first, if failed, automatically fallback to the referrers tag schema.

There might be a compatibility issue with some registries such as GAR.

Basically, attempting the Referrers API to GAR will not return a 404 but an unexpected 302.

See discussions:

priteshbandi · 2023-05-10T04:17:34Z

@Two-Hearts

For Sign:
Adding a new flag, --use-referrers-api (name TBD), behind experimental:
When both NOTATION_EXPERIMENTAL and --use-referrers-api are set, Notation would try the Referrers API first, if failed, automatically fallback to referrers tag schema.
Otherwise, Sign will be using referrers tag schema as default

IMO we should not fallback for sign operation i.e. When both NOTATION_EXPERIMENTAL and --use-referrers-api are set, Notation would try the Referrers API only. If failed, fail the sign operation and ask user to not use --use-referrers-api flag.

@shizhMSFT
IMO automatic fallback would useful for most of the registries in OCI 1.0 to 1.1 migration usecase. For scenarios where registries are misbehaving not abiding by spec we can have --not-use-referrers-api(name TBD) flag.

Two-Hearts · 2023-05-10T05:34:56Z

IMO we should not fallback for sign operation i.e. When both NOTATION_EXPERIMENTAL and --use-referrers-api are set, Notation would try the Referrers API only. If failed, fail the sign operation and ask user to not use --use-referrers-api flag.

Yeah, this also works. In this way, the Sign command becomes more deterministic (without any automatic fallback). Waiting for others' comments on this one.

Two-Hearts · 2023-05-10T05:49:28Z

Always try the Referrers API first, if failed, automatically fallback to the referrers tag schema.

There might be a compatibility issue with some registries such as GAR.

Basically, attempting the Referrers API to GAR will not return a 404 but an unexpected 302.

See discussions:

https://cloud-native.slack.com/archives/CQUH8U287/p1681159105868649

Introduce an experimental flag to enable backward compatibility with OCI registries #630

@shizhMSFT hmm, yes, we need to resolve issue 630. Do we want to have it done in RC.5 or v1.0?

yizha1 · 2023-05-10T06:46:46Z

IMO we should not fallback for sign operation i.e. When both NOTATION_EXPERIMENTAL and --use-referrers-api are set, Notation would try the Referrers API only. If failed, fail the sign operation and ask user to not use --use-referrers-api flag.

Yeah, this also works. In this way, the Sign command becomes more deterministic (without any automatic fallback). Waiting for others' comments on this one.

Yes, it makes sense. Use the experimental flag means you want to leverage Referrer API, and not tag schema. Otherwise, it may succeed with tag schema, which is not what users expected.

yizha1 · 2023-05-10T06:49:34Z

There might be a compatibility issue with some registries such as GAR.

Basically, attempting the Referrers API to GAR will not return a 404 but an unexpected 302.

See discussions:

https://cloud-native.slack.com/archives/CQUH8U287/p1681159105868649

Introduce an experimental flag to enable backward compatibility with OCI registries #630

Maybe we should raise it to OCI spec maintainers on the determination of referrer API

shizhMSFT · 2023-05-11T05:46:59Z

In order to make notation stable while we are building notation on top of the instable distribution-spec, we need to identify which part of the distribution-spec is relatively stable and which is not. Apparently, the referrers tag schema is relatively stable and should be used as the default behavior of notation. Meanwhile, the referrers API and its fallback strategy is instable and should be marked as experimental.

Based on the above fundamentals, I'd like to propose a new experimental flag --allow-referrers-api to all registry related commands such as notation sign and notation verify.

When --allow-referrers-api is toggled with NOTATION_EXPERIMENTAL=1, notation attempts the referrers API and fallback to referrers tag schema on failure.
Otherwise, notation has deterministic behavior and uses referrers tag schema.

From previous discussions, we have consensus on using referrers tag schema by default for notation sign but not aligned on notation verify. To me, notation verify should not attempt referrers API without NOTATION_EXPERIMENTAL=1 since the referrers API itself is instable / experimental.

Two-Hearts · 2023-05-11T06:09:34Z

In order to make notation stable while we are building notation on top of the instable distribution-spec, we need to identify which part of the distribution-spec is relatively stable and which is not. Apparently, the referrers tag schema is relatively stable and should be used as the default behavior of notation. Meanwhile, the referrers API and its fallback strategy is instable and should be marked as experimental.

Based on the above fundamentals, I'd like to propose a new experimental flag --allow-referrers-api to all registry related commands such as notation sign and notation verify.

When --allow-referrers-api is toggled with NOTATION_EXPERIMENTAL=1, notation attempts the referrers API and fallback to referrers tag schema on failure.

Otherwise, notation has deterministic behavior and uses referrers tag schema.

From previous discussions, we have consensus on using referrers tag schema by default for notation sign but not aligned on notation verify. To me, notation verify should not attempt referrers API without NOTATION_EXPERIMENTAL=1 since the referrers API itself is instable / experimental.

Yeah, after discussion with Shiwei offline, I agree with the above design. The automatic fallback from referrers API to tag schema is a part of the Referrers API's spec. As long as a user attempts the Referrers API, the process should include the fallback mechanism to align with the spec. From this point of view, I think we can still claim that Notation is deterministic regarding the use of Referrers API and Referrers tag schema.
@priteshbandi What do you think?

FeynmanZhou · 2023-05-11T08:08:29Z

@shizhMSFT 's proposal makes sense to me. Using a new experimental flag --allow-referrers-api to all registry-related commands will provide a consistent experience for signing and verification.

My only concern is that this is a breaking change for users still using v1.0.0-RC.4 with Referrers API by default. They have to add this additional experimental flag --allow-referrers-api when verifying their legacy signatures after upgrading to v1.0.0-RC.4. We may need to document the considerations in the v1.0.0-RC.5 release note and elaborate on related docs.
@priteshbandi @Two-Hearts @yizha1 @sajayantony

priteshbandi · 2023-05-12T22:19:49Z

@patrickzheng200 I agree, to summarize

If experimental flag is disabled (NOTATION_EXPERIMENTAL=0), dont use referrers API i.e.

For sign, only use referrers tag schema to store signatures.
For verify/inspect/list only use referrers tag schema to pull signatures.

If experimental flag is enabled (NOTATION_EXPERIMENTAL=1) and user uses `--use-referrers` flag; use oci 1.1 spec i.e. use referrers API

For sign, Check if referrers API returns 2xx or not. ~~If it doesnt returns 2xx fail signing because registry hasnt correctly/fully implemented oci1.1. If it return 2xx then sign using referrers.~~
- If referrers API returns 2xx, sign using referrers
- If referrers API returns 404, fallback or the OCI-Subject header was not set, fallback and use tag schema
- If referrers API returns anything else, fail the operation
For verify/inspect/list list, etc use referrers API first to pull the signature.
- If referrers API returns 2xx use it to list signatures
- If referrers API returns 404 or the OCI-Subject header was not set, fallback and use tag schema to list signatures
- If referrers API returns anything else, fail the operation

@FeynmanZhou Had similar discussion with @toddysm on Thursday, IMO we should mark RC4 as deprecated ASAP and ask customers to directly jump to rc5 from rc3.

cc: @iamsamirzon

Two-Hearts · 2023-05-15T02:21:32Z

If experimental flag is enabled (NOTATION_EXPERIMENTAL=1) and user uses --use-referrers flag; use oci 1.1 spec i.e. use referrers API

For sign, Check if referrers API returns 2xx or not. If it doesnt returns 2xx fail signing because registry hasnt correctly/fully implemented oci1.1. If it return 2xx then sign using referrers.

Thanks @priteshbandi, everything else LGTM. My only question is, when NOTATION_EXPERIMENTAL=1 and uses --allow-referrers flag during Sign, should we still enable fallback to referrers tag schema if referrers API is not supported? Because the fallback mechanism is a part of the Referrers API spec, trying referrers API without enabling the fallback schema seems not strictly following the distribution spec?
Quoting from the spec: If the [referrers API](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#listing-referrers) returns a 404, the client MUST fallback to pulling the [referrers tag schema](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#referrers-tag-schema).
Reference: https://github.com/opencontainers/distribution-spec/blob/main/spec.md#listing-referrers
/cc: @shizhMSFT

priteshbandi · 2023-05-15T04:10:41Z

My interpretation (I am not closely following spec) of spec is that fallback is only for pull operation but since sign is an push operation its better to provide consistency . Also, if user is opting in for oci spec 1.1 we should always use referrers API related behaviors. Want to hear others opinion.

yizha1 · 2023-05-15T05:02:39Z

My interpretation (I am not closely following spec) of spec is that fallback is only for pull operation but since sign is an push operation its better to provide consistency . Also, if user is opting in for oci spec 1.1 we should always use referrers API related behaviors. Want to hear others opinion.

@priteshbandi The pushing manifest procedure is defined here https://github.com/opencontainers/distribution-spec/blob/v1.1.0-rc2/spec.md#pushing-manifests-with-subject

priteshbandi · 2023-05-15T19:48:00Z

Thanks @Two-Hearts and @yizha1 for pointing to the right spec. As oci1.1-rc2 calls out, when user adds NOTATION_EXPERIMENTAL=1 and uses --allow-referrers flag, notation should perform fallback for sign operation.

Updated #659 (comment)

In this PR: 1. Removed Notation's support of signing with OCI artifact manifest. (One can still consume signatures in this type.) 2. Moved the Referrers API behind experimental as discussed in community meeting. This change takes effect on `sign`, `list`, `inspect`, and `verify` commands. When a user wants to try the Referrers API, they need to set both `NOTATION_EXPERIMENTAL=1` AND `--allow-referrers-api`; if the Referrers API is not supported, fallback to the Referrers tag schema automatically. Otherwise, by default, Notation would use the Referrers tag schema in all commands deterministically. References: https://github.com/opencontainers/distribution-spec/blob/v1.1.0-rc1/spec.md#listing-referrers https://github.com/opencontainers/distribution-spec/blob/v1.1.0-rc1/spec.md#referrers-tag-schema This PR has been tested and: resolves #659 resolves #660 --------- Signed-off-by: patrick <[email protected]> Signed-off-by: Patrick Zheng <[email protected]>

yizha1 added bug Something isn't working triage Need to triage labels May 9, 2023

yizha1 added this to the 1.0.0 milestone May 9, 2023

yizha1 removed the triage Need to triage label May 9, 2023

priteshbandi mentioned this issue May 10, 2023

Release v1.0.0-rc.5 for notation #661

Closed

4 tasks

priteshbandi mentioned this issue May 10, 2023

Deprecate --signature-manifest flag from notation sign #660

Closed

Two-Hearts mentioned this issue May 12, 2023

update: align Notation with OCI specs #663

Merged

yizha1 modified the milestones: 1.0.0, v1.0.0-rc.5 May 15, 2023

yizha1 assigned patrickzheng200 May 15, 2023

priteshbandi closed this as completed in #663 May 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mark feature "support OCI Referrer API" as experimental feature #659

Mark feature "support OCI Referrer API" as experimental feature #659

yizha1 commented May 9, 2023

yizha1 commented May 9, 2023

Two-Hearts commented May 9, 2023

Two-Hearts commented May 9, 2023

yizha1 commented May 9, 2023

Two-Hearts commented May 9, 2023

shizhMSFT commented May 9, 2023 •

edited

Loading

iamsamirzon commented May 9, 2023

priteshbandi commented May 9, 2023 •

edited

Loading

yizha1 commented May 10, 2023 •

edited

Loading

For Sign:

priteshbandi commented May 10, 2023 •

edited

Loading

Two-Hearts commented May 10, 2023 •

edited

Loading

shizhMSFT commented May 10, 2023 •

edited

Loading

priteshbandi commented May 10, 2023 •

edited

Loading

Two-Hearts commented May 10, 2023

Two-Hearts commented May 10, 2023 •

edited

Loading

yizha1 commented May 10, 2023

yizha1 commented May 10, 2023 •

edited

Loading

shizhMSFT commented May 11, 2023

Two-Hearts commented May 11, 2023

FeynmanZhou commented May 11, 2023

priteshbandi commented May 12, 2023 •

edited

Loading

Two-Hearts commented May 15, 2023

If experimental flag is enabled (NOTATION_EXPERIMENTAL=1) and user uses `--use-referrers` flag; use oci 1.1 spec i.e. use referrers API

priteshbandi commented May 15, 2023 •

edited

Loading

yizha1 commented May 15, 2023 •

edited

Loading

priteshbandi commented May 15, 2023 •

edited

Loading

Mark feature "support OCI Referrer API" as experimental feature #659

Mark feature "support OCI Referrer API" as experimental feature #659

Comments

yizha1 commented May 9, 2023

What is the areas you experience the issue in?

What is not working as expected?

What did you expect to happen?

How can we reproduce it?

Describe your environment

What is the version of your Notation CLI or Notation Library?

yizha1 commented May 9, 2023

Two-Hearts commented May 9, 2023

Two-Hearts commented May 9, 2023

yizha1 commented May 9, 2023

Two-Hearts commented May 9, 2023

shizhMSFT commented May 9, 2023 • edited Loading

iamsamirzon commented May 9, 2023

priteshbandi commented May 9, 2023 • edited Loading

For Sign:

For Verify, inspect, list aka any command that only read's content from registry:

yizha1 commented May 10, 2023 • edited Loading

For Sign:

priteshbandi commented May 10, 2023 • edited Loading

Two-Hearts commented May 10, 2023 • edited Loading

shizhMSFT commented May 10, 2023 • edited Loading

priteshbandi commented May 10, 2023 • edited Loading

Two-Hearts commented May 10, 2023

Two-Hearts commented May 10, 2023 • edited Loading

yizha1 commented May 10, 2023

yizha1 commented May 10, 2023 • edited Loading

shizhMSFT commented May 11, 2023

Two-Hearts commented May 11, 2023

FeynmanZhou commented May 11, 2023

priteshbandi commented May 12, 2023 • edited Loading

If experimental flag is disabled (NOTATION_EXPERIMENTAL=0), dont use referrers API i.e.

If experimental flag is enabled (NOTATION_EXPERIMENTAL=1) and user uses --use-referrers flag; use oci 1.1 spec i.e. use referrers API

Two-Hearts commented May 15, 2023

If experimental flag is enabled (NOTATION_EXPERIMENTAL=1) and user uses --use-referrers flag; use oci 1.1 spec i.e. use referrers API

priteshbandi commented May 15, 2023 • edited Loading

yizha1 commented May 15, 2023 • edited Loading

priteshbandi commented May 15, 2023 • edited Loading

shizhMSFT commented May 9, 2023 •

edited

Loading

priteshbandi commented May 9, 2023 •

edited

Loading

yizha1 commented May 10, 2023 •

edited

Loading

priteshbandi commented May 10, 2023 •

edited

Loading

Two-Hearts commented May 10, 2023 •

edited

Loading

shizhMSFT commented May 10, 2023 •

edited

Loading

priteshbandi commented May 10, 2023 •

edited

Loading

Two-Hearts commented May 10, 2023 •

edited

Loading

yizha1 commented May 10, 2023 •

edited

Loading

priteshbandi commented May 12, 2023 •

edited

Loading

If experimental flag is enabled (NOTATION_EXPERIMENTAL=1) and user uses `--use-referrers` flag; use oci 1.1 spec i.e. use referrers API

If experimental flag is enabled (NOTATION_EXPERIMENTAL=1) and user uses `--use-referrers` flag; use oci 1.1 spec i.e. use referrers API

priteshbandi commented May 15, 2023 •

edited

Loading

yizha1 commented May 15, 2023 •

edited

Loading

priteshbandi commented May 15, 2023 •

edited

Loading