feat!: add referrersHandler for artifact manifest #34

wangxiaoxuan273 · 2022-11-10T03:51:52Z

This pr implements referrersHandler and its related operations. For this pr only artifact manifest referrers are supported. Support for image manifest referrers will be added in a future pr. The implementation is based on the existing rc2 branch, which implements referrersHandler as an extension. Changes the registry/handler package.

Part 7 of #21.

wangxiaoxuan273 · 2022-11-10T05:16:17Z

I have run a basic test.

Setting: subject.json is a minimal artifact manifest. ref01.json, ref02.json, ref03.json are artifact manifests that refer to subject.json.
Test 1: Get the referrers of subject.json. An image index with a manifest list of length 3 should be returned.
Test 2: Get the referrers of ref03.json, which has no referrers. An image index with an empty manifest list should be returned.

Test result:

Test 1:

Prettified returned result:

{
	"schemaVersion": 2,
	"mediaType": "application/vnd.oci.image.index.v1+json",
	"manifests": [{
		"mediaType": "application/vnd.oci.artifact.manifest.v1+json",
		"digest": "sha256:6696649520b4354ecb926244ea2cebe3fa5ad4f109c1f15f317b50da75b3f92c",
		"size": 392,
		"annotations": {
			"tick": "tock",
			"woo": "doo"
		},
		"artifactType": "application/referrer2"
	}, {
		"mediaType": "application/vnd.oci.artifact.manifest.v1+json",
		"digest": "sha256:b971dd46783d9f43a1aba9018cfa10125a23e5fdb47fab252f6919f85439d811",
		"size": 325,
		"artifactType": "application/referrerexample"
	}, {
		"mediaType": "application/vnd.oci.artifact.manifest.v1+json",
		"digest": "sha256:cd07ace460801b6441c6aa1d9a738cc37514da66553f4257048ad7f59948535b",
		"size": 406,
		"annotations": {
			"bar": "bbaarr",
			"foo": "ffoooo"
		},
		"artifactType": "application/vnd.example.sbom.v1"
	}]
}

Test 2:

codecov-commenter · 2022-11-10T05:29:47Z

Codecov Report

Merging #34 (96ff32f) into main (ad70be1) will decrease coverage by 0.68%.
The diff coverage is 1.47%.

@@            Coverage Diff             @@
##             main      #34      +/-   ##
==========================================
- Coverage   57.16%   56.48%   -0.69%     
==========================================
  Files         107      108       +1     
  Lines       11041    11176     +135     
==========================================
+ Hits         6312     6313       +1     
- Misses       4019     4153     +134     
  Partials      710      710

Impacted Files	Coverage Δ
registry/handlers/manifests.go	`53.18% <ø> (ø)`
registry/handlers/referrers.go	`0.00% <0.00%> (ø)`
registry/storage/paths.go	`69.31% <25.00%> (-1.12%)`	⬇️
registry/handlers/app.go	`47.08% <100.00%> (+0.07%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

m5i-work · 2022-11-10T07:46:07Z

registry/handlers/referrers.go

+	v1 "github.com/opencontainers/image-spec/specs-go/v1"
+)
+
+const createAnnotationName = "io.cncf.oras.artifact.created"


Should this be OCI version?

Changed to "org.opencontainers.artifact.created".

m5i-work · 2022-11-10T07:48:21Z

registry/handlers/referrers.go

+				return err
+			}
+
+			artifactManifest, ok := man.(*ociartifact.DeserializedManifest)


Do we want to cover OCI image manifest referrers in this PR?

We would do that in a separate pr. For this pr we only covers OCI artifact manifest. Thanks for pointing this out.

m5i-work · 2022-11-10T07:49:01Z

registry/handlers/referrers.go

+	}
+
+	w.Header().Set("Content-Type", v1.MediaTypeImageIndex)
+	w.Header().Set("ORAS-Api-Version", "oras/1.0")


Is this still relevant?

removed it.

m5i-work · 2022-11-10T07:51:47Z

registry/handlers/referrers.go

+}
+
+func generateLinkHeader(repoName, subjectDigest, artifactType string, lastDigests []string, nPage int) string {
+	url := fmt.Sprintf("/v2/%s/_oras/artifacts/referrers?digest=%s&nextToken=%s",


The url needs to be updated

updated this url.

shizhMSFT · 2022-11-11T07:35:39Z

@wangxiaoxuan273 Could you split this PR into two PRs? One for indexing the newly pushed OCI Artifact manifest. One for query the referrers API?

wangxiaoxuan273 · 2022-11-18T08:08:06Z

registry/handlers/referrers.go

+	v1 "github.com/opencontainers/image-spec/specs-go/v1"
+)
+
+const createAnnotationName = "org.opencontainers.artifact.created"


@shizhMSFT Is this variable okay? I didn't find any related info about this in the spec.

Try searching org.opencontainers.artifact.created in https://github.com/opencontainers/distribution-spec/blob/main/spec.md

Note: createAnnotationName for Artifact manifest and image manifest are different.

Found relevant information in this page. Added doc comments in the code.

wangxiaoxuan273 · 2022-11-18T08:12:30Z

registry/handlers/referrers.go

+}
+
+func generateLinkHeader(repoName, subjectDigest, artifactType string, lastDigests []string, nPage int) string {
+	url := fmt.Sprintf("/v2/%s/_referrers/manifests/referrers?digest=%s&nextToken=%s",


@shizhMSFT Is this url okay?

The URL is defined in https://github.com/opencontainers/distribution-spec/blob/main/spec.md#listing-referrers

There is no such a field called nextToken.

Therefore, nextToken is an implementation specific field. You might want to document this somewhere.

I think we may not need the nextToken field. I've created an issue to keep track of the paging feature of referrers, and I'll look deeper into it when fixing that issue.

Updated the url according to the distribution spec.

shizhMSFT · 2022-11-21T05:01:42Z

registry/storage/paths.go

+var (
+	rootPrefix = []string{storagePathRoot, storagePathVersion}
+	repoPrefix = append(rootPrefix, "repositories")
+)


You may want to put it back as local variables.

Think about the following code.

storagePathVersion := "v2" storagePathRoot := "/docker/registry/" rootPrefix := []string{storagePathRoot, storagePathVersion} repoPrefix := append(rootPrefix, "repositories") pathA := append(repoPrefix, "foo") pathB := append(repoPrefix, "bar") fmt.Println(pathA) fmt.Println(pathB)

The output is

[/docker/registry/ v2 repositories bar] [/docker/registry/ v2 repositories bar]

Put it back.

shizhMSFT · 2022-11-21T05:02:28Z

registry/handlers/referrers.go

+// Reference for pre-defined annotation keys:
+// https://github.com/opencontainers/image-spec/blob/main/annotations.md#pre-defined-annotation-keys
+const createAnnotationNameArtifactManifest = "org.opencontainers.artifact.created"
+const createAnnotationTimestampFormat = time.RFC3339
+const maxPageSize = 100
+
+// minimum page size used for # of digests to put in nextToken
+const minPageSize = 3


You may want to make them const blocks.

shizhMSFT · 2022-11-21T05:04:18Z

registry/handlers/referrers.go

+	artifactType := r.FormValue("artifactType")
+	nPage, nParseError := strconv.Atoi(r.FormValue("n"))


Do FormValue apply to GET requests?

replaced FormValue with r.URL.Query()

shizhMSFT · 2022-11-21T05:06:43Z

registry/handlers/referrers.go

+	}
+}
+
+func (h *referrersHandler) generateSortedReferrers(ctx context.Context, revision digest.Digest, artifactType string) ([]v1.Descriptor, error) {


Does OCI Referrer Spec specifies sorting? If so, please point out the spec reference.

Removed sorting

m5i-work · 2022-11-23T06:38:19Z

registry/handlers/referrers.go

+func referrersDispatcher(ctx *Context, r *http.Request) http.Handler {
+	dgst, err := getDigest(ctx)
+	if err != nil {
+


nit: extra blank line?

m5i-work · 2022-11-23T06:43:41Z

registry/handlers/referrers.go

+}
+
+// ReferrersAPIResponse describes the response body of the referrers API.
+type ReferrersAPIResponse struct {


Looks like we are still missing annotations field?

Btw, can we reuse v1.Index?

Agree that v1.Index looks better.

Yes, I forgot annotations. Reused v1.Index here instead and added annotations.

m5i-work · 2022-11-23T06:46:55Z

registry/handlers/referrers.go

+	}
+
+	nextTokens := query["nextToken"]
+	TokenSet := make(map[string]string)


Why does this var start with capital letter?

This variable is from the original rc2 branch implementation. Removed it now as it is for the paging feature, and paging will be implemented in a separate pr.

m5i-work · 2022-11-23T07:12:25Z

registry/handlers/referrers.go

+}
+
+func generateLinkHeader(repoName, subjectDigest, artifactType string, lastDigests []string, nPage int) string {
+	url := fmt.Sprintf("/v2/%s/referrers/%s?nextToken=%s",


url.Values() may help here.

removed this function as it is for paging and paging will be implemented in another pr.

shizhMSFT · 2022-11-23T16:20:54Z

registry/handlers/referrers.go

+	dcontext.GetLogger(h).Debug("GetReferrers")
+
+	query := r.URL.Query()
+	n, nExists := query["n"]


Should use Values.Get() method to avoid array boundary issues.

Replaced with Get().

shizhMSFT · 2022-11-23T16:21:44Z

registry/handlers/referrers.go

+}
+
+// ReferrersAPIResponse describes the response body of the referrers API.
+type ReferrersAPIResponse struct {


Agree that v1.Index looks better.

shizhMSFT · 2022-11-23T16:23:43Z

registry/handlers/referrers.go

+	}
+
+	nextTokens := query["nextToken"]
+	TokenSet := make(map[string]string)


Since it is a set, value is never used.

Suggested change

TokenSet := make(map[string]string)

tokenSet := make(map[string]struct{})

removed this part as it is part of the paging feature.

shizhMSFT · 2022-11-23T16:28:58Z

registry/handlers/referrers.go

+func enumerateReferrerLinks(ctx context.Context,
+	rootPath string,
+	stDriver driver.StorageDriver,
+	repository distribution.Repository,
+	blobstatter distribution.BlobStatter,
+	subjectRevision digest.Digest,
+	artifactManifestIndex map[digest.Digest][]digest.Digest,
+	ingestor func(ctx context.Context,
+		digest digest.Digest,
+		subjectRevision digest.Digest,
+		artifactManifestIndex map[digest.Digest][]digest.Digest,
+		repository distribution.Repository,
+		blobstatter distribution.BlobStatter,
+		storageDriver driver.StorageDriver) error) error {


This function declaration does not seem right.

It can be simplified as

Suggested change

func enumerateReferrerLinks(ctx context.Context,

rootPath string,

stDriver driver.StorageDriver,

repository distribution.Repository,

blobstatter distribution.BlobStatter,

subjectRevision digest.Digest,

artifactManifestIndex map[digest.Digest][]digest.Digest,

ingestor func(ctx context.Context,

digest digest.Digest,

subjectRevision digest.Digest,

artifactManifestIndex map[digest.Digest][]digest.Digest,

repository distribution.Repository,

blobstatter distribution.BlobStatter,

storageDriver driver.StorageDriver) error) error {

func enumerateReferrerLinks(ctx context.Context,

rootPath string,

stDriver driver.StorageDriver,

blobstatter distribution.BlobStatter,

ingestor func(digest digest.Digest) error) error {

Changed accordingly. Thanks for the work.

shizhMSFT · 2022-11-23T16:31:39Z

registry/handlers/referrers.go

+		err = ingestor(ctx,
+			digest,
+			subjectRevision,
+			artifactManifestIndex,
+			repository,
+			blobstatter,
+			stDriver)
+		if err != nil {
+			return err
+		}
+
+		return nil


When you see the pattern

err = do() if err != nil { return err } return nil

it can be simplified as

return do()

changed accordingly.

shizhMSFT · 2022-11-23T16:40:01Z

registry/handlers/referrers.go

+	// only consider pagination if # of referrers is greater than page size
+	if len(referrers) > nPage {
+		startIndex := 0
+		if len(TokenSet) > 0 {
+			for i, ref := range referrers {
+				// check if ref matches a digest in nextToken list
+				if _, ok := TokenSet[ref.Digest.String()]; ok {
+					// set the starting index to the largest index
+					if (i + 1) > startIndex {
+						startIndex = i + 1
+					}
+				}
+			}
+		}
+
+		// only applicable if last item provided as nextToken
+		if startIndex == len(referrers) {
+			referrers = []v1.Descriptor{}
+		}
+
+		// if there's only 1 page of results left
+		if len(referrers)-startIndex <= nPage {
+			referrers = referrers[startIndex:]
+		} else {
+			referrers = referrers[startIndex:(startIndex + nPage)]
+			// nextToken is a base64 encoded comma-separated string of the digests of the last three referrers in the response
+			var nextDgsts []string
+			for i := nPage - 1; i >= nPage-minPageSize; i-- {
+				nextDgsts = append(nextDgsts, referrers[i].Digest.String())
+			}
+			// if n was not provided in page, set nPage to a value so link header knows not to include n
+			if nParseError != nil {
+				nPage = -1
+			}
+			// add the Link Header
+			w.Header().Set("Link", generateLinkHeader(h.Repository.Named().Name(), h.Digest.String(), filterArtifactType, nextDgsts, nPage))
+		}
+	}


The next token implementation is strange to me. Could you explain why you choose this algorithm?

Removed the paging part, will do it in a future pr.

Signed-off-by: wangxiaoxuan273 <[email protected]>

shizhMSFT · 2022-11-25T08:25:25Z

registry/handlers/referrers.go

+	linked, err := digest.Parse(string(content))
+	if err != nil {
+		return "", err
+	}
+
+	return linked, nil


code pattern can be applied.

Suggested change

linked, err := digest.Parse(string(content))

if err != nil {

return "", err

}

return linked, nil

return digest.Parse(string(content))

changed accordingly.

shizhMSFT · 2022-11-25T08:31:44Z

registry/handlers/referrers.go

+	annotations := map[string]string{}
+	query := r.URL.Query()
+	artifactTypeFilter := query.Get("artifactType")
+	if artifactTypeFilter != "" {
+		annotations[v1.AnnotationReferrersFiltersApplied] = "artifactType"
+	}


note that nil map with omitempty will be omitted. However, empty map will be encoded to {}.

Here, if artifactType is not there, we should use nil map.

Consider this code pattern

Suggested change

annotations := map[string]string{}

query := r.URL.Query()

artifactTypeFilter := query.Get("artifactType")

if artifactTypeFilter != "" {

annotations[v1.AnnotationReferrersFiltersApplied] = "artifactType"

}

var annotations map[string]string

artifactTypeFilter := r.URL.Query().Get("artifactType")

if artifactTypeFilter != "" {

annotations = map[string]string{

v1.AnnotationReferrersFiltersApplied: "artifactType",

}

}

changed accordingly.

shizhMSFT · 2022-11-25T08:40:23Z

registry/handlers/referrers.go

+
+func (h *referrersHandler) generateReferrersList(ctx context.Context, subjectDigest digest.Digest, artifactType string) ([]v1.Descriptor, error) {
+	dcontext.GetLogger(ctx).Debug("(*referrersHandler).generateReferrersList")
+	var referrers []v1.Descriptor


nit: code style: let's only declare variables when needed.
suggestion: move this line after line 99 but before line 102

moved accordingly.

Signed-off-by: wangxiaoxuan273 <[email protected]>

shizhMSFT

LGTM

wangxiaoxuan273 mentioned this pull request Nov 10, 2022

Add OCI Artifact Support #21

Open

m5i-work reviewed Nov 10, 2022

View reviewed changes

wangxiaoxuan273 commented Nov 18, 2022

View reviewed changes

wangxiaoxuan273 changed the title ~~feat!: add referrersHandler~~ feat!: add referrersHandler that works for artifact manifest Nov 21, 2022

wangxiaoxuan273 requested a review from shizhMSFT November 21, 2022 04:52

shizhMSFT reviewed Nov 21, 2022

View reviewed changes

wangxiaoxuan273 requested review from shizhMSFT and m5i-work and removed request for shizhMSFT November 22, 2022 05:00

m5i-work reviewed Nov 23, 2022

View reviewed changes

shizhMSFT requested changes Nov 23, 2022

View reviewed changes

wangxiaoxuan273 added 3 commits November 25, 2022 07:30

combined commits for rebase

ed6a559

Signed-off-by: wangxiaoxuan273 <[email protected]>

fixed error in enuerate link

9aeb405

Signed-off-by: wangxiaoxuan273 <[email protected]>

removed dead code

fe2494e

Signed-off-by: wangxiaoxuan273 <[email protected]>

wangxiaoxuan273 requested a review from shizhMSFT November 25, 2022 07:59

wangxiaoxuan273 changed the title ~~feat!: add referrersHandler that works for artifact manifest~~ feat!: add referrersHandler for artifact manifest Nov 25, 2022

shizhMSFT reviewed Nov 25, 2022

View reviewed changes

resolved comments

96ff32f

Signed-off-by: wangxiaoxuan273 <[email protected]>

shizhMSFT approved these changes Nov 25, 2022

View reviewed changes

shizhMSFT merged commit 9ac1115 into oras-project:main Nov 25, 2022

wangxiaoxuan273 deleted the referrers-handler branch November 26, 2022 11:47

wangxiaoxuan273 mentioned this pull request Jan 4, 2023

bug: referrers should return an empty manifest list, but it now returns null #51

Closed

		artifactType := r.FormValue("artifactType")
		nPage, nParseError := strconv.Atoi(r.FormValue("n"))

	TokenSet := make(map[string]string)
	tokenSet := make(map[string]struct{})

feat!: add referrersHandler for artifact manifest #34

feat!: add referrersHandler for artifact manifest #34

Conversation

wangxiaoxuan273 commented Nov 10, 2022 • edited Loading

wangxiaoxuan273 commented Nov 10, 2022 • edited Loading

codecov-commenter commented Nov 10, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangxiaoxuan273 Nov 18, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shizhMSFT commented Nov 11, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shizhMSFT Nov 18, 2022 • edited Loading

Choose a reason for hiding this comment

wangxiaoxuan273 Nov 21, 2022 • edited Loading

Choose a reason for hiding this comment

wangxiaoxuan273 Nov 18, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangxiaoxuan273 Nov 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangxiaoxuan273 Nov 25, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shizhMSFT Nov 25, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shizhMSFT left a comment

Choose a reason for hiding this comment

wangxiaoxuan273 commented Nov 10, 2022 •

edited

Loading

wangxiaoxuan273 commented Nov 10, 2022 •

edited

Loading

codecov-commenter commented Nov 10, 2022 •

edited

Loading

wangxiaoxuan273 Nov 18, 2022 •

edited

Loading

shizhMSFT Nov 18, 2022 •

edited

Loading

wangxiaoxuan273 Nov 21, 2022 •

edited

Loading

wangxiaoxuan273 Nov 18, 2022 •

edited

Loading

wangxiaoxuan273 Nov 21, 2022 •

edited

Loading

wangxiaoxuan273 Nov 25, 2022 •

edited

Loading

shizhMSFT Nov 25, 2022 •

edited

Loading