Specify Params in Pipeline

[bkuschel]

Expected Behavior

We want to be able to provide Pipeline wide parameters which can be used in templating across Tasks.

For example, I should be able to create something like this (spec for params can be different!)

kind: Pipeline

...
spec:
  params:
    - name: 'namespace'
      value: 'test'
      type: 'string' # this is assuming we go ahead with something like #207 and add some types
- name: deploy-bundle-test
  taskRef:
    name: deploy-with-kubectl
  inputSourceBindings:
  - name: workspace
    resourceRef:
        name: guestbook    
  params:
  - name: namespace
    value: "${params.namespace}" # this should come from the global params

Actual Behavior

PipelineRun only accepts Resources, not parameters. We can specify templating within a Pipeline, but it only references Resources within the Task that uses the templating, there is no concept of global templating.

Additional Info

For example, have a pipeline with a helm deployment step that has a parameter that specifies the namespace to install into (for isolation.)

Strongly related to: #254

Note that one strange thing here is that:

• At the Task level, we express parameters we want in Tasks, and we specify the params we're actually using within the `TaskRun
• At the Pipeline level, the params exist within the Pipeline only, and are never provided in the PipelineRun

In the future we may want to consider moving more params into PipelineRuns??? Maybe!

[bobcatfish]

Hey @bkuschel ! Thanks for opening this issue :D

I think we may have this functionality already, though let me know if this isn't what you're looking for.

You could have a helm task like this:

kind: Task
metadata:
  name: deploy-with-helm
  namespace: default
spec:
  inputs:
    resources:
    - name: workspace
      type: git
...
    params:
    - name: namespace
...
  steps:
  - name: deploy
    image: kubernetes-helm
    command: ['helm']
    args:
      - 'install'
      - '--namespace'
      - '${inputs.params.namespace}'

Then you could use templating in the Pipeline to provide the value of the namespace parameter:

kind: Pipeline
...
- name: deploy-bundle-test
  taskRef:
    name: deploy-with-kubectl
  inputSourceBindings:
  - name: workspace
    resourceRef:
        name: guestbook    
  params:
  - name: namespace
    value: "${inputs.resources.workspace.revision}"

(From #64)

But you can only use inputs and outputs from the Task in the templating - I'm thinking what you want is some kind of "global" parameter (like in #254) that you can use for the namespace and pass into Tasks throughout the Pipeline?

[bobcatfish]

(Note: if we do want a "pipeline wide variable" I'm thinking that evolving PipelineParams like you suggested in slack is probably the way to go @bkuschel !)

[abayer]

Yeah, I know my first example in #254 was based on the idea of Pipeline parameters that can be referenced by all the Tasks contained in the Pipeline, so we don't have to duplicate setting Task parameters multiple times.

[bkuschel]

But you can only use inputs and outputs from the Task in the templating - I'm thinking what you want is some kind of "global" parameter

@bobcatfish right, something that is PipelineRun specific (ie, pertains only to a particular run of the pipeline).

[bobcatfish]

Cool, sounds good to me! Anyone have any other ideas besides extending PipelineParams to contain arbitrary key/values? (might tie in well with #121!!)

[bobcatfish]

Another idea would be to have a field in the Pipeline itself (I guess we don't have to use PipelineParams just because it exists XD)

[bobcatfish]

Another idea would be to have a field in the Pipeline itself (I guess we don't have to use PipelineParams just because it exists XD)

In #282 we are getting rid of PipelineParams (as we know it) so adding these global vars to Pipelines probably makes the most sense atm :D

[bobcatfish]

@bkuschel I updated the description pretty significantly, let me know (or update it!) if this no longer expresses what you're looking for

[bkuschel]

yes, pipeline is good but I was thinking of being able to provide this through PipelineRun as well. Here's the specific scenario..

1. I create a common pipeline with tasks for the basic logic of deploying some composition of microservices that constitutes some app that is be to deployed to several environments (ie. clusters/namespaces). Environments are distinct in terms of back-end connections, domains, secrets, etc.
2. The pipeline has tasks with steps that have been generalized so that theses environment configurations can be supplied through code.
3. One option is to provide this configuration as part of github repo resources input into the task but as some steps are configured to use pre-fab docker images ("builders") from other providers, some configurations need to be passed in as environment variables to the pods

So at this point, if I am to fire off a pipelinerun for a particular environment i'd need to pass in the unique repositories for the environment as well as the parameters that eventually make their way down to the steps as environmental variables. If this is only provided through pipeline, i'd need to redefine the pipeline for every run which makes it difficult to separate out the commonality between environments.

One idea i had was to create a resource that is just a bag of properties that's passed around.

I hope that makes sense.

[bobcatfish]

(sorry for the super delayed reply @bkuschel , this shouldn't be the norm!)

One idea i had was to create a resource that is just a bag of properties that's passed around.

Interesting suggestion - in earlier iterations of the Pipeline we had something called PipelineParams which I think would have satisfied the use case you're describing here.

It'd be interesting to see if we can find a design for something similar that works with our latest CRDs.

I'm planning to create an issue in the near future talking about how event triggering systems would create new PipelineRuns - in the current design they end up needing to make a lot of PipelineResources as well, but if we had some kind of "parameters" type, it could be possible to provide overrides instead 🤔

[abayer]

So my initial thinking on this is that we probably need to have the ability to specify parameters themselves (with options for a default value for the parameter or if a value for the parameter is required, perhaps) in Pipeline, and values for those parameters in PipelineRun. Then the parameter values would be used at runtime via PipelineTask to pass to the actual task execution, and would also be available for use in Pipeline itself for values in things like Timeout and any future Pipeline configuration fields (like Pipeline-wide environment variables, i.e., #254). I'll start work on a PR shortly.

[abayer]

/assign abayer

[bobcatfish]

Thanks @abayer !! 🎉

[abayer]

I'm still wondering if there's anything else we want to allow parameter values to be used in, but for now, it really seemed like PipelineTask parameters were the only one that made sense. Of course, once we tackle #254, there'll be something else. =)