Towards tektoncd project with no dependencies to plumbing

[vdemeester]

Related to tektoncd/triggers#180 (comment) — use go modules for tektoncd/triggers. This issue aims to discuss and track a way to make so that tektoncd projects do not need explicit dependency to tektoncd/plumbing.

hm interesting! I think this highlights something very odd (and historical) about our plumbing repo - we were using go to vendor in bash scripts basically thinking (is there anything besides the bash scripts we need from plumbing?)

(My experiences with git submodules in the past were painful but I can't remember the specifics so maybe it's okay?)

I think no matter what we should work toward not sharing scripts from plumbing like this, so this is more motivation to migrate from the bash scripts to some combo of

• Tekton Tasks (shared via one of @vdemeester 's catalog designs? tektoncd/pipeline#964)
• Code (I thought python made a lot of sense but then we're going to have to start publishing python packages? maybe sticking to Go makes more sense)
• Images

Couple ideas:

1. Use git submodule for now
2. Update whatever actually needs the plumbing scripts to fetch them (probably our .sh e2e scripts)
3. Hold off on this until we migrate away from the bash scripts

I'm partial to (2) tho it's more work but would settle for (1) (can always do 2 as a next step :))

Today, most of the projects (tektoncd/pipeline, tektoncd/cli, …) use a small hack with dep or go mod to get a specific version of pluming scripts as a go dependency. As @bobcatfish highlight, this highlight an historical odd thing with plumbing (inherited from knative).

We should aim towards a tektoncd/plumbing dependency-free. Having all project using tekton pipelines is the end goal and would make those projects standing on their own, tektoncd/plumbing would be the infra setup and cross project CI/CD if any. That said, as discussed on the issue with @bobcatfish, we should do this in several step, ensuring all project are doing the same (easier to follow and adapt).

Here are my initial proposals

1. For now, either keep with the hack or use git submodules
   • tektoncd/experimental is using something different too, so we are gonna need to change that in any case
   • Please shout if you don't want git submodules 😝. I (and @bobcatfish too) have little experience with them in the past few years so, not sure what to think about it yet.
2. Bake the current script in the test-runner image used for CI (with prow), so that we don't need to have any as dependency. This would force us to use script only for CI and make sure anything can run locally without those scripts (and document this)
3. Use Tekton to build Tekton projects, using our own pipeline-as-code as convention (nothing official at first, just the convention we use/experiment with in tektoncd organization).

/cc @bobcatfish @dibyom @skaegi @chmouel @mnuttall @imjasonh @vtereso @afrittoli @dlorenc @sbwsg

[chmouel]

I would worry about git submodules I can't point exactly why but I have the feeling we will be exchanging a set of issues with another..

I think 2. is a reasonable mid-term fix to our issue, that's not too hard to implement.

Solution 3. is a rather attractive long term solution but I guess we will need to speak about it more before getting there.

Just to mention it, while I was working on OpenStack, they had this openstack-common (oslo it was called) thing that was imported/updated and committed directly into projects. It wasn't pretty and had its own set of issues but it was rather working for a while.....
until it growed too big and they had to broke down all those common code to dedicated libraries shipped on pypi.

[afrittoli]

I think (3) is the way to go, I don't see the need for intermediate steps.

We need some design about how we are going to share plumbing tasks (and Tekton resources in general) to the other projects.

We also need to decide / design whether the entry point of CI jobs will stay in plumbing or if it will in each repo or both.

It would be nice if the model we adopt for plumbing embodied the recommended setup that organisations can use to define their own tekton based CI/CD.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.