TEP-0005: Tekton OCI Bundle #137

vdemeester · 2020-06-24T15:47:04Z

This proposal is to be able to bundle Tasks (and Pipelines, and Resources, and potentially other future config objects) into an OCI Artifact, pushed to an image registry, and referenced from that registry.

We need to agree on a name too (cc @wlynch for the bundle part that was discussed in a complete different context 😛)

This is a early version, based on

/cc @imjasonh @pierretasci

Signed-off-by: Vincent Demeester [email protected]

tekton-robot · 2020-06-24T15:47:06Z

@vdemeester: GitHub didn't allow me to request PR reviews from the following users: pierretasci.

Note that only tektoncd members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

This proposal is to be able to bundle Tasks (and Pipelines, and Resources, and potentially other future config objects) into an OCI Artifact, pushed to an image registry, and referenced from that registry.

We need to agree on a name too (cc @wlynch for the bundle part that was discussed in a complete different context 😛)

This is a early version, based on

Tekton OCI Image Catalog

Tekton OCI Image Design

/cc @imjasonh @pierretasci

Signed-off-by: Vincent Demeester [email protected]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

imjasonh · 2020-06-24T18:37:51Z

Link to rendered Markdown: https://github.com/vdemeester/tektoncd-community/blob/tep-tekton-oci-bundles/teps/0003-tekton-oci-bundles.md

bobcatfish · 2020-06-24T20:26:30Z

Link to rendered Markdown: https://github.com/vdemeester/tektoncd-community/blob/tep-tekton-oci-bundles/teps/0003-tekton-oci-bundles.md

btw @imjasonh you can also hit this button to see the rendered version in the files view (still cant comment tho):

vdemeester · 2020-06-25T05:52:09Z

btw, https://hackmd.io/RPvSTjA_RSuVOu2FEmj8bQ is where I edit this file 😉 (and you can edit it with me, comment, …)

jlpettersson · 2020-07-03T11:29:37Z

teps/0003-tekton-oci-bundles.md

+### Goals
+
+The goal of this TEP is to define the specification for the Tekton OCI
+bundles. The goal of the Tekton OCI bundles is to define a Spec on top


Just a Spec? and not a full resource?

The distinction is actually really important. They key use case that OCI Bundles unlocks is the ability to refer to Tasks/Pipelines outside of the cluster. This allows multiple definitions of the same-named Task/Pipeline to exist and for users to version them. Therefore, the metadata of the resource is unimportant. It is really the spec we are after as we wish to dynamically insert that at runtime.

It means that we again will get the same problem as this PR was intended to fix tektoncd/pipeline#2826

By providing the full resource rather than a part of a resource the end-user has the possibility to add labels and annotations. But this was mostly a question - it may not be relevant - but it is hard to reason since the full context (e.g. use) is not specified here.

The Spec here should be rewritten as specification, it doesn't have anything to do with the spec field of a Tekton Resource 😅 😉

I don't think the purpose of a TEP is to get too involved into the implementation but just to entertain this...there is nothing stopping the implementation of this from propagating the metadata of the task. It is possible either way (though to be clear, my preference is to not propagate labels and annotations from tasks to the pod).

jlpettersson · 2020-07-03T11:31:31Z

teps/0003-tekton-oci-bundles.md

+
+### Non-Goals
+
+How Tekton OCI bundles are used (referred to) in the different Tekton


I feel that it is hard to reason about this feature - e.g. hard to understand what it really is - when there is no explanation about how these are used.

For more background, the original design doc might be useful: https://docs.google.com/document/d/1lXF_SvLwl6OqqGy8JbpSXRj4hWJ6CSImlxlIl4V9rnM/edit

jlpettersson · 2020-07-03T11:37:12Z

teps/0003-tekton-oci-bundles.md

+
+Our most important capability is enabling references of remote Tasks
+from within TaskRuns or Pipelines and Pipelines from within
+PipelineRuns.


I struggle to understand this. E.g. a new PipelineRun is created, but it contains a reference to a Pipeline declared within an image. So the controller - instead of accessing a resource through informers and via ApiServer and etcd - it should now start to pull images from a registry - or multiple different registries - then extract data from the images (I imaging this is a kind of unzipping or similar) and then - no caching? - so next time the run is reconciled - the same procedure is happening? This is unclear and confusing for me - with the information that is provided so far.

What we define here is merely the contract so the implementation details such as, is the image cached, are intentionally left out. But to your earlier sentiment, that is indeed correct. Rather than use the Kubernetes API (which is how Tekton looks up references at the moment), we would like to enable external references. This solves use cases of versioning, idempotency, etc.

I need to update the doc with some user stories and usage example. As @pierretasci the scope of this TEP is only to define the specification of the OCI artifact that would "transport" the tekton definitions. We can add examples of how we think they could be used by the CLI, the pipeline controller, etc… but it's out of the scope here.
I'll add those examples next week 😉

So the controller - instead of accessing a resource through informers and via ApiServer and etcd - it should now start to pull images from a registry - or multiple different registries - then extract data from the images (I imaging this is a kind of unzipping or similar) and then - no caching? - so next time the run is reconciled - the same procedure is happening? This is unclear and confusing for me - with the information that is provided so far.

This will be on the controller side to decide those things (caching, etc..). And this, most likely, will be an enhancement over this TEP or a new TEP dedicated on that.

pierretasci

LGTM, I think this captures the contract as originally intended by the design.

imjasonh

/lgtm 🎉

I'd like to see some more details about how access to the bundles is authorized, but since integration into Pipelines is out of scope for this doc we can just punt the auth issue until then.

imjasonh · 2020-07-22T12:18:53Z

teps/0005-tekton-oci-bundles.md

+of this file) is [here](/teps/NNNN-TEP-template/README.md).
+
+-->
+# TEP-NNNN: Tekton OCI bundles


This can be 0005 now

vdemeester · 2020-07-22T12:24:52Z

I'd like to see some more details about how access to the bundles is authorized, but since integration into Pipelines is out of scope for this doc we can just punt the auth issue until then.

Right, I didn't want to go too deep into this. That said, we could say that one of the benefit of this proposal is that we can re-use what already exists for OCI images.

dlorenc · 2020-07-24T19:58:51Z

teps/0005-tekton-oci-bundles.md

+  in-cluster functionality. You wouldn't be able to store two Tekton
+  Tasks of the same `name` and `apiVersion` in a namespace so to make
+  the task references easier to reason about, we enforce the same
+  characteristic on the images.


How is this enforced? I don't think registries will do anything to prevent it...

You are right that registries will not enforce this. Likely, we would enforce this on pull. In particular because we rely on these annotations to understand what resource is in each layer, if it isn't specified, it is easy to reject the OCI image as it will be unusable.

Makes sense! Another approach might be to treat these layers just like an actual image layers, and have the last one win...

Not sure which is better. I like the idea of not imposing extra semantics on what can be a valid bundle, but this is probably a rare case anyway.

dlorenc · 2020-07-24T19:59:15Z

teps/0005-tekton-oci-bundles.md

+
+- An image will store each resource as a new layer
+  This allows us to quickly store and retrieve individual resources
+  because we can key the layer they reside in against its metadatax


nit: metadata (no x) :)

dlorenc · 2020-07-24T19:59:43Z

teps/0005-tekton-oci-bundles.md

+this later but we can still do all of the same things with the default
+`MIME` type. Nothing prevents us for adding a new custom `MIME` type
+in the future when we are confident that most registries supports
+custom mime types.


+1 to this, support just isn't there yet.

dlorenc · 2020-07-24T20:00:43Z

A few nits/questions, otherwise this looks good!

dlorenc · 2020-07-24T20:00:58Z

/lgtm

Signed-off-by: Vincent Demeester <[email protected]>

afrittoli

Thanks for this!
/approve

tekton-robot · 2020-07-31T15:49:08Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: afrittoli, pierretasci

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~teps/OWNERS~~ [afrittoli]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

imjasonh · 2020-07-31T17:01:46Z

teps/0005-tekton-oci-bundles.md

+
+- Using OCI images and layer for what they are not intended for — at
+least with the usual `MIME` type — might be confusing for users. A
+`docker pull gcr.io/tekton-catalog/task/golang-build:0.1` will fail


At a minimum, we should document the error message you'll see when you docker pull it, so that users can find it and learn what it means (e.g., "You tried to pull a Tekton OCI bundle, which is not runnable").

Going further, do we have any say in how it fails? For instance, we could document/require (and support with tooling) that the image should actually have a valid MIME type, and when it's docker run, it will print some useful error message? Do helm images do anything like this?

imjasonh · 2020-07-31T17:02:24Z

One minor non-blocking comment. Otherwise lgtm! 🎉

/lgtm

afrittoli · 2020-08-13T15:39:38Z

@vdemeester I failed to notice that this TEP does not have the header part. Could you fix that as a follow-up?

Signed-off-by: Priya Wadhwa <[email protected]> Signed-off-by: Priya Wadhwa <[email protected]>

tekton-robot requested a review from imjasonh June 24, 2020 15:47

tekton-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jun 24, 2020

jlpettersson reviewed Jul 3, 2020

View reviewed changes

pierretasci approved these changes Jul 3, 2020

View reviewed changes

vdemeester mentioned this pull request Jul 16, 2020

Reorganize Catalog based on Catalog Organization proposal tektoncd/catalog#386

Closed

vdemeester force-pushed the tep-tekton-oci-bundles branch from 5f96456 to 082453e Compare July 22, 2020 11:03

vdemeester changed the title ~~TEP-000X: Tekton OCI Bundle~~ TEP-0005: Tekton OCI Bundle Jul 22, 2020

imjasonh reviewed Jul 22, 2020

View reviewed changes

vdemeester force-pushed the tep-tekton-oci-bundles branch from 082453e to 9a70e32 Compare July 22, 2020 12:25

bobcatfish mentioned this pull request Jul 23, 2020

Design: Sharing Tasks and Pipelines without Copy Pasta tektoncd/catalog#45

Closed

dlorenc reviewed Jul 24, 2020

View reviewed changes

tekton-robot assigned dlorenc Jul 24, 2020

tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Jul 24, 2020

TEP-0005 - Tekton OCI Bundle TEP

077ede1

Signed-off-by: Vincent Demeester <[email protected]>

vdemeester force-pushed the tep-tekton-oci-bundles branch from 9a70e32 to 077ede1 Compare July 31, 2020 15:40

tekton-robot removed the lgtm Indicates that a PR is ready to be merged. label Jul 31, 2020

afrittoli reviewed Jul 31, 2020

View reviewed changes

tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 31, 2020

imjasonh reviewed Jul 31, 2020

View reviewed changes

tekton-robot assigned imjasonh Jul 31, 2020

tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Jul 31, 2020

tekton-robot merged commit e9659c5 into tektoncd:master Jul 31, 2020

vdemeester deleted the tep-tekton-oci-bundles branch July 31, 2020 17:44

imjasonh mentioned this pull request Aug 17, 2020

TEP-0012: API Spec #172

Merged

dlorenc pushed a commit to dlorenc/community that referenced this pull request Oct 27, 2022

Add sigstore oncall team (tektoncd#137)

a6762ab

Signed-off-by: Priya Wadhwa <[email protected]> Signed-off-by: Priya Wadhwa <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TEP-0005: Tekton OCI Bundle #137

TEP-0005: Tekton OCI Bundle #137

vdemeester commented Jun 24, 2020

tekton-robot commented Jun 24, 2020

imjasonh commented Jun 24, 2020

bobcatfish commented Jun 24, 2020

vdemeester commented Jun 25, 2020

jlpettersson Jul 3, 2020

pierretasci Jul 3, 2020

jlpettersson Jul 3, 2020

vdemeester Jul 3, 2020

pierretasci Jul 3, 2020

jlpettersson Jul 3, 2020

pierretasci Jul 3, 2020

jlpettersson Jul 3, 2020

pierretasci Jul 3, 2020

vdemeester Jul 3, 2020 •

edited

Loading

pierretasci left a comment

imjasonh left a comment

imjasonh Jul 22, 2020

vdemeester commented Jul 22, 2020

dlorenc Jul 24, 2020

pierretasci Jul 24, 2020

dlorenc Jul 24, 2020

dlorenc Jul 24, 2020

dlorenc Jul 24, 2020

dlorenc commented Jul 24, 2020

dlorenc commented Jul 24, 2020

afrittoli left a comment

tekton-robot commented Jul 31, 2020

imjasonh Jul 31, 2020

imjasonh commented Jul 31, 2020

afrittoli commented Aug 13, 2020


		### Non-Goals

		How Tekton OCI bundles are used (referred to) in the different Tekton

TEP-0005: Tekton OCI Bundle #137

TEP-0005: Tekton OCI Bundle #137

Conversation

vdemeester commented Jun 24, 2020

tekton-robot commented Jun 24, 2020

imjasonh commented Jun 24, 2020

bobcatfish commented Jun 24, 2020

vdemeester commented Jun 25, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vdemeester Jul 3, 2020 • edited Loading

Choose a reason for hiding this comment

pierretasci left a comment

Choose a reason for hiding this comment

imjasonh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vdemeester commented Jul 22, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dlorenc commented Jul 24, 2020

dlorenc commented Jul 24, 2020

afrittoli left a comment

Choose a reason for hiding this comment

tekton-robot commented Jul 31, 2020

Choose a reason for hiding this comment

imjasonh commented Jul 31, 2020

afrittoli commented Aug 13, 2020

vdemeester Jul 3, 2020 •

edited

Loading