Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kind/feat: Surface artifacts through termination message #7714

Conversation

ericzzzzzzz
Copy link
Contributor

@ericzzzzzzz ericzzzzzzz commented Feb 25, 2024

Fixes: #7717

  • In is part of implementation of TEP-0147
  • allow users write artifacts in step, artifacts will be surfaced to TaskRun status stepState.

Changes

  • resolve $(step.artifacts.path) to /tekton/steps/<container-name>/artifacts/provenance.json in the Step container at runtime, and users can write artifacts provenance data to the path. Details about the data structure https://github.com/tektoncd/community/blob/main/teps/0147-tekton-artifacts-phase1.md#artifact-provenance-data
  • surface artifacts through termination message, provenance data is written to termination message and populates to stepState.
  • support consuming provenance output data with syntax -
    • $(steps.<the-name-of-step>.outputs) --> outputs are artifact values from the first artifact
    • $(steps.<the-name-of-step>.outputs.<artifact-name>) --> outputs for a specific artifact
    • $(steps.<the-name-of-step>.inputs) --> inputs are artifact values from the first artifact
    • $(steps.<the-name-of-step>.inputs.<artifact-name>) --> inputs for a specific artifact
  • add validation and use feature flag to guard artifacts feature
    • interpolations with $(step.artifacts.path) and $(steps.<step-name>.outputs/inputs.<artifact-category-name>) can only be used in step.Script, step.Env, step.Args, steps.Common with enable-artifacts feature is set to true
    • Note that, if $(steps.<step-name>.outputs/inputs.<artifact-category-name>) is used, we will read the script file and re-write container command with script.

Submitter Checklist

As the author of this PR, please check off the items in this checklist:

  • Has Docs if any changes are user facing, including updates to minimum requirements e.g. Kubernetes version bumps
  • Has Tests included if any functionality added or changed
  • pre-commit Passed
  • Follows the commit message standard
  • Meets the Tekton contributor standards (including functionality, content, code)
  • Has a kind label. You can add one by adding a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
  • Release notes block below has been updated with any user facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings). See some examples of good release notes.
  • Release notes contains the string "action required" if the change requires additional action from users switching to the new release

Release Notes

Surface artifact metadata through termination message

/kind feat

@tekton-robot
Copy link
Collaborator

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@tekton-robot tekton-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Feb 25, 2024
@tekton-robot tekton-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Feb 25, 2024
@ericzzzzzzz
Copy link
Contributor Author

/kind feat

@tekton-robot
Copy link
Collaborator

@ericzzzzzzz: The label(s) kind/feat cannot be applied, because the repository doesn't have them.

In response to this:

/kind feat

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ericzzzzzzz
Copy link
Contributor Author

/kind feature

@tekton-robot tekton-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Feb 25, 2024
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1beta1/taskrun_conversion.go 91.9% 92.7% 0.8
pkg/entrypoint/entrypointer.go 89.0% 87.5% -1.5
pkg/pod/status.go 95.2% 95.0% -0.2
pkg/reconciler/taskrun/resources/apply.go 99.5% 99.6% 0.0
pkg/reconciler/taskrun/taskrun.go 87.2% 87.2% 0.0
pkg/result/result.go 86.7% 81.2% -5.4

@ericzzzzzzz ericzzzzzzz force-pushed the surface-artifacts-through-termination-message branch from 16ed163 to 7443e91 Compare March 11, 2024 16:56
@tekton-robot tekton-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Mar 11, 2024
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/entrypoint/entrypointer.go 89.0% 87.5% -1.5
pkg/reconciler/taskrun/resources/apply.go 99.5% 99.5% 0.0
pkg/reconciler/taskrun/taskrun.go 87.2% 87.2% 0.0
pkg/result/result.go 86.7% 81.2% -5.4

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/entrypoint/entrypointer.go 89.0% 87.5% -1.5
pkg/reconciler/taskrun/resources/apply.go 99.5% 99.5% 0.0
pkg/reconciler/taskrun/taskrun.go 87.2% 87.2% 0.0
pkg/result/result.go 86.7% 81.2% -5.4

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/entrypoint/entrypointer.go 89.0% 68.0% -20.9
pkg/reconciler/taskrun/resources/apply.go 99.5% 99.5% 0.0
pkg/reconciler/taskrun/taskrun.go 87.2% 87.2% 0.0
pkg/result/result.go 86.7% 81.2% -5.4

@tekton-robot tekton-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 13, 2024
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/entrypoint/entrypointer.go 89.0% 88.6% -0.4
pkg/reconciler/taskrun/resources/apply.go 99.5% 99.5% 0.0
pkg/reconciler/taskrun/taskrun.go 87.2% 87.2% 0.0
pkg/result/result.go 86.7% 81.2% -5.4

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/entrypoint/entrypointer.go 89.0% 88.6% -0.4
pkg/reconciler/taskrun/resources/apply.go 99.5% 99.5% 0.0
pkg/reconciler/taskrun/taskrun.go 87.2% 87.2% 0.0
pkg/result/result.go 86.7% 81.2% -5.4

@ericzzzzzzz ericzzzzzzz changed the title kind/feat: Surface artifacts through termination message kind/feat: Surface artifacts through termination message Mar 13, 2024
@tekton-robot tekton-robot added release-note-none Denotes a PR that doesnt merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Mar 13, 2024
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/entrypoint/entrypointer.go 89.0% 88.2% -0.8
pkg/pod/status.go 95.2% 95.0% -0.2
pkg/reconciler/taskrun/resources/apply.go 99.5% 99.5% 0.0
pkg/reconciler/taskrun/taskrun.go 87.2% 87.2% 0.0
pkg/result/result.go 86.7% 81.2% -5.4

@tekton-robot tekton-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesnt merit a release note. labels Mar 13, 2024
@ericzzzzzzz
Copy link
Contributor Author

Some questions about the pr:

  • Do I need to include some md files for this feature at the moment ?
  • Do I need to include some examples to show how to write/consume artifacts provenance data

@ericzzzzzzz ericzzzzzzz marked this pull request as ready for review March 13, 2024 15:48
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/task_validation.go 98.5% 98.6% 0.1
pkg/apis/pipeline/v1beta1/task_validation.go 98.4% 98.6% 0.1
pkg/entrypoint/entrypointer.go 89.0% 86.7% -2.3
pkg/pod/status.go 95.2% 95.0% -0.2
pkg/reconciler/taskrun/resources/apply.go 99.5% 99.5% 0.0
pkg/reconciler/taskrun/taskrun.go 87.2% 87.2% 0.0
pkg/result/result.go 86.7% 87.5% 0.8

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/task_validation.go 98.5% 98.6% 0.1
pkg/apis/pipeline/v1beta1/task_validation.go 98.4% 98.6% 0.1
pkg/entrypoint/entrypointer.go 89.0% 86.7% -2.3
pkg/pod/status.go 95.2% 95.0% -0.2
pkg/reconciler/taskrun/resources/apply.go 99.5% 99.5% 0.0
pkg/reconciler/taskrun/taskrun.go 87.2% 87.2% 0.0
pkg/result/result.go 86.7% 87.5% 0.8

@ericzzzzzzz ericzzzzzzz force-pushed the surface-artifacts-through-termination-message branch from b8abe73 to 1ea4d6e Compare April 5, 2024 19:06
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/task_validation.go 98.5% 98.6% 0.1
pkg/apis/pipeline/v1beta1/task_validation.go 98.4% 98.6% 0.1
pkg/entrypoint/entrypointer.go 89.0% 86.7% -2.3
pkg/pod/status.go 95.2% 95.0% -0.2
pkg/reconciler/taskrun/resources/apply.go 99.5% 99.5% 0.0
pkg/reconciler/taskrun/taskrun.go 87.2% 87.2% 0.0
pkg/result/result.go 86.7% 87.5% 0.8

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/task_validation.go 98.5% 98.6% 0.1
pkg/apis/pipeline/v1beta1/task_validation.go 98.4% 98.6% 0.1
pkg/entrypoint/entrypointer.go 89.0% 86.7% -2.3
pkg/pod/status.go 95.2% 95.0% -0.2
pkg/reconciler/taskrun/resources/apply.go 99.5% 99.5% 0.0
pkg/reconciler/taskrun/taskrun.go 87.2% 87.2% 0.0
pkg/result/result.go 86.7% 87.5% 0.8

@tekton-robot tekton-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 6, 2024
@ericzzzzzzz ericzzzzzzz force-pushed the surface-artifacts-through-termination-message branch from 1ea4d6e to 285f5e6 Compare April 6, 2024 14:50
@tekton-robot tekton-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 6, 2024
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/task_validation.go 98.5% 98.6% 0.1
pkg/apis/pipeline/v1beta1/task_validation.go 98.4% 98.6% 0.1
pkg/pod/status.go 95.2% 95.0% -0.2
pkg/reconciler/taskrun/resources/apply.go 99.5% 99.5% 0.0
pkg/reconciler/taskrun/taskrun.go 87.2% 87.2% 0.0
pkg/result/result.go 86.7% 87.5% 0.8

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/task_validation.go 98.5% 98.6% 0.1
pkg/apis/pipeline/v1beta1/task_validation.go 98.4% 98.6% 0.1
pkg/pod/status.go 95.2% 95.0% -0.2
pkg/reconciler/taskrun/resources/apply.go 99.5% 99.5% 0.0
pkg/reconciler/taskrun/taskrun.go 87.2% 87.2% 0.0
pkg/result/result.go 86.7% 87.5% 0.8

@ericzzzzzzz ericzzzzzzz force-pushed the surface-artifacts-through-termination-message branch from 285f5e6 to de45f72 Compare April 6, 2024 18:48
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/task_validation.go 98.5% 98.6% 0.1
pkg/apis/pipeline/v1beta1/task_validation.go 98.4% 98.6% 0.1
pkg/entrypoint/entrypointer.go 89.0% 86.7% -2.3
pkg/pod/status.go 95.2% 95.0% -0.2
pkg/reconciler/taskrun/resources/apply.go 99.5% 99.5% 0.0
pkg/reconciler/taskrun/taskrun.go 87.2% 87.2% 0.0
pkg/result/result.go 86.7% 87.5% 0.8

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/task_validation.go 98.5% 98.6% 0.1
pkg/apis/pipeline/v1beta1/task_validation.go 98.4% 98.6% 0.1
pkg/entrypoint/entrypointer.go 89.0% 86.7% -2.3
pkg/pod/status.go 95.2% 95.0% -0.2
pkg/reconciler/taskrun/resources/apply.go 99.5% 99.5% 0.0
pkg/reconciler/taskrun/taskrun.go 87.2% 87.2% 0.0
pkg/result/result.go 86.7% 87.5% 0.8

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/task_validation.go 98.5% 98.6% 0.1
pkg/apis/pipeline/v1beta1/task_validation.go 98.4% 98.6% 0.1
pkg/entrypoint/entrypointer.go 89.0% 78.9% -10.1
pkg/pod/status.go 95.2% 95.0% -0.2
pkg/reconciler/taskrun/resources/apply.go 99.5% 99.5% 0.0
pkg/reconciler/taskrun/taskrun.go 87.2% 87.2% 0.0
pkg/result/result.go 86.7% 87.5% 0.8

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/task_validation.go 98.5% 98.6% 0.1
pkg/apis/pipeline/v1beta1/task_validation.go 98.4% 98.6% 0.1
pkg/entrypoint/entrypointer.go 89.0% 78.9% -10.1
pkg/pod/status.go 95.2% 95.0% -0.2
pkg/reconciler/taskrun/resources/apply.go 99.5% 99.5% 0.0
pkg/reconciler/taskrun/taskrun.go 87.2% 87.2% 0.0
pkg/result/result.go 86.7% 87.5% 0.8

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/task_validation.go 98.5% 98.6% 0.1
pkg/apis/pipeline/v1beta1/task_validation.go 98.4% 98.6% 0.1
pkg/entrypoint/entrypointer.go 89.0% 86.7% -2.3
pkg/pod/status.go 95.2% 95.0% -0.2
pkg/reconciler/taskrun/resources/apply.go 99.5% 99.5% 0.0
pkg/reconciler/taskrun/taskrun.go 87.2% 87.2% 0.0
pkg/result/result.go 86.7% 87.5% 0.8

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/task_validation.go 98.5% 98.6% 0.1
pkg/apis/pipeline/v1beta1/task_validation.go 98.4% 98.6% 0.1
pkg/entrypoint/entrypointer.go 89.0% 86.7% -2.3
pkg/pod/status.go 95.2% 95.0% -0.2
pkg/reconciler/taskrun/resources/apply.go 99.5% 99.5% 0.0
pkg/reconciler/taskrun/taskrun.go 87.2% 87.2% 0.0
pkg/result/result.go 86.7% 87.5% 0.8

@ericzzzzzzz
Copy link
Contributor Author

hi @afrittoli I updated the pr according your suggestions! Please take another look, thank you for the review !

@ericzzzzzzz ericzzzzzzz force-pushed the surface-artifacts-through-termination-message branch from 7b0282b to 7e2768c Compare April 10, 2024 17:14
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/task_validation.go 98.5% 98.6% 0.1
pkg/apis/pipeline/v1beta1/task_validation.go 98.4% 98.6% 0.1
pkg/entrypoint/entrypointer.go 89.0% 86.7% -2.3
pkg/pod/status.go 95.2% 95.0% -0.2
pkg/reconciler/taskrun/resources/apply.go 99.5% 99.5% 0.0
pkg/reconciler/taskrun/taskrun.go 87.2% 87.2% 0.0
pkg/result/result.go 86.7% 87.5% 0.8

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage-df to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1/task_validation.go 98.5% 98.6% 0.1
pkg/apis/pipeline/v1beta1/task_validation.go 98.4% 98.6% 0.1
pkg/entrypoint/entrypointer.go 89.0% 86.7% -2.3
pkg/pod/status.go 95.2% 95.0% -0.2
pkg/reconciler/taskrun/resources/apply.go 99.5% 99.5% 0.0
pkg/reconciler/taskrun/taskrun.go 87.2% 87.2% 0.0
pkg/result/result.go 86.7% 87.5% 0.8

Comment on lines +49 to +55
- name: artifacts-consumer-python
image: python:latest
script: |
#!/usr/bin/env python3
import json
data = json.loads('$(steps.artifacts-producer.outputs)')
print(data[0]['uri'])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Comment on lines +615 to +630
dataBytes, err := os.ReadFile(e.Command[0])
if err != nil {
return err
}
fileContent := string(dataBytes)
v, err := replaceValue(artifactref.StepArtifactRegex, fileContent, stepDir, getArtifactValues)
if err != nil {
return err
}
if v != fileContent {
temp, err := writeToTempFile(v)
if err != nil {
return err
}
e.Command = []string{temp.Name()}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the coverage report, this section is not well covered by unit tests, but we have E2E coverage for this, so it should be fine.

Copy link
Member

@afrittoli afrittoli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for all the updates, it looks good to me.
I only have one comment, about one logging line, nothing worth blocking this PR one.
Would you mind addressing that in a follow-up PR?
/lgtm

@@ -281,6 +282,15 @@ func setTaskRunStatusBasedOnStepStatus(ctx context.Context, logger *zap.SugaredL
logger.Errorf("termination message could not be parsed as JSON: %v", err)
merr = multierror.Append(merr, err)
} else {
for _, r := range results {
if r.ResultType == result.ArtifactsResultType {
if err := json.Unmarshal([]byte(r.Value), &as); err != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: in all other error cases, we have a logger.Errorf message, should we have one here too?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure I'll do it !

@tekton-robot tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Apr 11, 2024
@tekton-robot tekton-robot merged commit 599f2e6 into tektoncd:main Apr 11, 2024
13 checks passed
@ericzzzzzzz ericzzzzzzz mentioned this pull request Apr 11, 2024
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. kind/feature Categorizes issue or PR as related to a new feature. lgtm Indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

TEP-0147 surface artifacts through termination message
5 participants