Implement Pipeline Templating

[bobcatfish]

Expected Behavior

When a user creates a Pipeline, they should be able to use templating to pass arguments to tasks, using the values that will eventually (via PipelineRun) be provided by the referenced PipelineParams and Resources.

This should include:

• Unit test coverage
• Docs on how to use this templating

Actual Behavior

Templating is being designed in #36 but is not implemented.

Steps to Reproduce the Problem

1. Deploy the Pipeline CRD + related CRDs to your cluster (note: should be docs on how to do this post kubebuilder)
2. Create a Task like this:

apiVersion: pipeline.knative.dev/v1beta1
kind: Task
metadata:
  name: template-params-task
  namespace: default
spec:
    inputs:
        params:
           - name: stuffToEcho
             value: string
    buildSpec:
        steps:
            - name: helloworld
              image: busybox
              args: ['echo', '${stuffToEcho}']

3. Create Resources and PipelineParams like this (we're going to use the clusters):

apiVersion: pipeline.knative.dev/v1beta1
kind: Resource
metadata:
  name: template-resources
  namespace: default
spec:
  resources:
  - name: github-resource
    type: git
    params:
    - name: url
      value: https://github.com/grafeas/kritis
    - name: revision
      value: 123456789123456789
  - name: image-resource
    type: image
    params:
    - name: url
      value: gcr.io/somethingsomethingsomething

apiVersion: pipeline.knative.dev/v1beta1
kind: PipelineParams
metadata:
  name: template-params
  namespace: default
spec:
    clusters:
        - name: 'testCluster'
          type: 'gke'
          endpoint: 'https://somethingsomethingsomethingsomething.com'

4. Create a Pipeline like this that uses templating to pass values from Resources and PipelineParams:

apiVersion: pipeline.knative.dev/v1beta1
kind: Pipeline
metadata:
  name: template-pipeline
  namespace: default
spec:
    tasks:
        - name: template-params-task-image
          taskRef:
              name: template-params-task
          params:
              - name: stuffToEcho
                value: ${resources.image-resource.url}
        - name: template-params-task-github
          taskRef:
              name: template-params-task
          params:
              - name: stuffToEcho
                value: ${resources.github-resource.revision}
        - name: template-params-task-cluster
          taskRef:
              name: template-params-task
          params:
              - name: stuffToEcho
                value: ${pipelineParams.testCluster.endpoint}

4. Create a PipelineRun like this:

apiVersion: pipeline.knative.dev/v1beta1
kind: PipelineRun
metadata:
  name: template-param-run-1
  namespace: default
spec:
    pipelineRef:
        name: template-pipeline
    resourceRef:
        name: template-resources
    pipelineParamsRef:
        name: template-params
    triggerRef:
        type: manual

5. Creating the PipelineRun should have created TaskRuns with all of the values filled in, so with kubectl get taskRuns you should be able to see:

• Conditions updated appropriately
• params (see this example) filled in with the templated values

6. Looking at the Task logs (if this has been implemented, see Implement working "hello world" Task + TaskRun #59) you should be able to see the appropriate values echoed for each task, i.e.:

• template-params-task-image should echo gcr.io/somethingsomethingsomething
• template-params-task-github should echo 123456789123456789
• template-params-task-cluster should echo https://somethingsomethingsomethingsomething.com

Additional Info

Blocked by #36 (design templating) and #61 (simple PipelineRun)

[bobcatfish]

This is the design doc for both templating and references, shared with members of knative-dev@.

[bobcatfish]

/assign @dlorenc
/meow space

[knative-prow-robot]

@bobcatfish:

In response to this:

/assign @dlorenc
/meow space

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[nader-ziada]

@bobcatfish should this issue be closed? I think its done now by the above PRs

[bobcatfish]

I think that actually templating with a Task's outputs still isn't implemented 🤔 But i think @dlorenc has a branch with this fixed so we can nearly close this...

[bobcatfish]

The other thing still missing is applying any templating at a Pipeline level.

[dlorenc]

Can this be closed now?

[bobcatfish]

Only use case left is when you provide templating via a Pipeline:

kind: Pipeline
...
- name: deploy-bundle-test
  taskRef:
    name: deploy-with-kubectl
  inputSourceBindings:
  - name: workspace
    key: guestbook    
  - name: myImage
    key: redis-docker
    passedConstraint:
    - build-push
  params:
  - name: labels
    value: "${myImage.digest}@${workspace.revision}"

In the end the value should be expanded

[dlorenc]

So I'm not 100% sure of the best way to test this. I added a test case to show that the PipelineRun reconciler does not expand these and will pass them to the TaskRun controller, and we have TaskRun tests that show they get expanded. But that doesn't feel very satisifying...

[abayer]

At first glance, at least, that sounds like a reasonable set of tests to me.

[bobcatfish]

I added a test case to show that the PipelineRun reconciler does not expand these and will pass them to the TaskRun controller, and we have TaskRun tests that show they get expanded. But that doesn't feel very satisifying...

I hear you! I can't think of anything better at the moment - we could have an integration tests that passes the whole thing in and makes sure the right thing comes out, but that sounds pretty rough.

I was looking at some of the kubebuilder stuff today with @pwittrock and the integration tests it generates, which can instantiate the controllers and api server locally, could be an interesting way to test something like this.

That being said it kinda feels like overkill!

TL;DR: @dlorenc I think either add the test you've described or just decide it's not worth covering and leave it as is.