[TEP-0146] Parameters in Script

[aaron-prindle]

This PR adds the problem statement for the TEP and identifies possible solutions. The proposal will be added in a subsequent PR after discussions of alternatives.

Using Parameter variables directly in script blocks in Tasks is a footgun in two ways:

• Security: It is easy for a Task author to accidentally introduce a vector for code injection and, by contrast, difficult for a Task user to verify that such an injection can't or hasn't taken place.
• Reliability: It is easy for a Task user to accidentally pass in a Parameter with a character that would make the Script invalid and fail the Task, making the Task extremely fragile.

To solve the above problems, this TEP aims to:

• Introduce a safe and reliable way to access Parameter variables from Scripts, and update the documentation and Tekton Catalog with the new approach.
• Disallow use of Parameter variables directly in script blocks of Steps in Tekton Pipelines V1 API.

References:

• Issues: * Provide shell-escaped parameters for script: pipeline#3226 * Escape incoming data triggers#675 * Use Environment Variables instead of Parameters in Script plumbing#971
• Catalog Guidance to Avoid Using Parameters in Script Blocks
• Tekton Enhancement Proposals: * TEP-0017: Shell-Escaped Parameters * TEP-0023: Implicit Parameters * TEP-0099: Parameters in Script

[jerop]

/kind tep

[vdemeester]

/assign

[JeromeJu]

/assign @Yongxuanzhang @JeromeJu

[tekton-robot]

@JeromeJu: GitHub didn't allow me to assign the following users: Yongxuanzhang.

Note that only tektoncd members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @Yongxuanzhang @JeromeJu

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Yongxuanzhang]

/assign

[QuanZhang-William]

Hi @aaron-prindle, thanks for the PR! We have a TEP template to leverage in case you don't know 😁

[afrittoli]

If we did fail validation, we would then basically not support them at all, which would be a backwards incompatible change. We could do that in v2, for v1 we might do that as an opt-in feature or a warning system - even though validation in itself does not have any facility for deprecations or warnings that I'm aware of, so it's unclear how the warning could be delivered to the user

[aaron-prindle]

+1, currently the idea for rolling this out would be to keep backwards compatibility with an opt-in feature-gate for cluster operators to remove this in v1 (if we agree this makes sense). For the warnings, it wasn’t clear to me the de-facto way to do this in Tekton. As stated, we could output some “WARN: Tekton is deprecating <...>”, see docs link here for more information on deprecation timelines how to migrate <doc-link AND/OR link to TBD bespoke tkn command for migration>” on creation of a Task but not sure how visible this might be to most end users.

[afrittoli]

Thanks for this proposal.
I left a couple of comments.

NIT: I have the impression that there was a proposal with the .env syntax initially documented, and then removed in favour of a list of alternatives, as some part of the document seem to assume a specific design and others don't.

In any case, I'm happy to have this merged as a proposal, but I'd like to see more coverage of array/object params as well as non-bash environments (e.g. python).

[aaron-prindle]

Thank you for your feedback @afrittoli & @vdemeester! I’ve updated the document addressing all of the feedback posted (see individual comments). Wanted to address the points in you final comment below:

• NIT: I have the impression that there was a proposal with the .env syntax initially documented, and then removed in favour of a list of alternatives, as some part of the document seem to assume a specific design and others don't.

  • Apologies for the confusion in the document, I have updated the document which was unclear in some places. Originally some options in the doc were very similar so I have deduped things now and no longer reference any “implicit” option as it was confusing and is not really an option due to backwards compat guarantees. The proposed solution here is broadly to add additional param suffixes (*.env, *.path, etc.) that modify how a value is interpolated (+ potentially other configuration) with the primary one (and first to be implemented) being *.env. Then we would gauge interest from reviewers on if we should add other options (*.path) or possibly what the preferred option would be if folks don’t agree on *.env as the highest priority.

• In any case, I'm happy to have this merged as a proposal, but I'd like to see more coverage of array/object params as well as non-bash environments (e.g. python).

  • Great, I’ve updated the TEP to explain how the array and object support would work and added examples across the preferred solution for array/object params. Currently the plan would be that the values in the env var map 1:1 with how they are currently used as that UX is working for users atm. In the case of an array param, it would be the same values based on the syntax, for an array param + index (params.arr[0]) we would get the string value and for the entire array we would provide the list as a comma separated array (params.arr -> first,second,third…) The TEP suggestion is built around the script field which only supports string type variables which currently has comma separated serialization for the array and actually has a validation error against using an object directly (vs an object field). I have added some examples under the “Suggested Solution”.
  • Additionally I’ve added some an example related to non-bash environments (eg: python), happy to elaborate in the document or add more examples - currently not sure the best way to enhance the doc related to this.

[aaron-prindle]

Thanks for the reviews here! @afrittoli - does this look good now to be merged as “proposed”? If not, what are the open questions to resolve to get this to implementable? I've added additional information and examples around array params, obj params, and non-bash environments. Happy to answer any additional q's or iterate on the doc as needed. Thanks!

[vdemeester]

What's the suggested solution here ? Usage of $(params.url.env) or having ${PARAMS_URL} working by default ? I am not sure I understand the before/after/resolved.

• Are we mutating the Task definition (to add the environment variables), or are we just setting them at Pod creation time ?
• What happens if params.url and params.URL exists ? (I think we error out on those already, but I am not 100% sure 😅 ).

[aaron-prindle]

Hi @vdemeester, responding to your questions below:

What's the suggested solution here ? Usage of $(params.url.env) or having ${PARAMS_URL} working by default ?

• Usage of $(params.url.env) is the recommended solution here,

Are we mutating the Task definition (to add the environment variables), or are we just setting them at Pod creation time ?

• This is somewhat of an open question that I am hoping to explore/discuss more in the "implementable" phase (happy to discuss it more now well, I'm not quite sure the cutoffs for the phase distinctions). The initial proposal was to set them in the Task definition (to add the environment variables) but
  the suggestion to set them @ Pod creation time above ([TEP-0146] Parameters in Script #1077 (comment)) is an alternative with pros which I plan on investigating after this is merged as proposed. To me this is an implementation detail IIUC, happy to continue discussing it here.

What happens if params.url and params.URL exists ? (I think we error out on those already, but I am not 100% sure 😅 ).

• This was mentioned in this comment as well - [TEP-0146] Parameters in Script #1077 (comment) (the translation from param name to env variable name risks being brittle). In testing this, tektoncd/pipeline allows this currently - you can successfully create a Pipeline with params.url and params.URL and the validation hook allows this. The allowable chars for env vars are: [A-Za-z0-9_]+. The options here would be:
  • add a sha suffix to the env var PARAM_*_<SHA_SUFFIX> based on the param name to guarantee uniqueness for param -> env var conversion.
  • use param variable names with the correct case, eg: PARAM_url and PARAM_URL

[vdemeester]

Sounds good, thanks @aaron-prindle
/approve

[tekton-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: afrittoli, JeromeJu, vdemeester

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• teps/OWNERS [afrittoli,vdemeester]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[aaron-prindle]

Thanks @vdemeester! Lmk if there is anything I need to do on my end to get this merged as proposed, currently seeing:
tide Pending — Not mergeable. Needs lgtm label.

[Yongxuanzhang]

We need someone who has the permission to lgtm

[Yongxuanzhang]

This tep is very similar like https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-an-intermediate-environment-variable ? Is it mentioned in the proposal as a reference?

[aaron-prindle]

This tep is very similar like https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-an-intermediate-environment-variable ? Is it mentioned in the proposal as a reference?

@Yongxuanzhang - just added this as a reference, thanks!

[dibyom]

/lgtm