Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create Run resources from PipelineTask references #3115

Closed
wants to merge 1 commit into from

Conversation

imjasonh
Copy link
Member

@imjasonh imjasonh commented Aug 19, 2020

Changes

Progress toward #3133

This change allows TaskRefs in PipelineTasks to refer to a non-Tekton ("custom") task type. When such a Pipeline is run, the controller will create a new Run resource that embeds that reference, at which point a custom task controller can take over and begin reconciling that execution, eventually to completion.

Followups for future PRs:

  • documentation about specifying Custom Task refs in a Pipeline
  • RunStatus is not yet surfaced in the PipelineRunStatus since it currently introduces an import cycle 💣 -- need to move RunStatus to v1beta1 even though Run is still in v1alpha1.

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

  • [y] Includes tests (if functionality changed/added)
  • [n] Includes docs (if user facing)
  • [y] Commit messages follow commit message best practices
  • [y] Release notes block has been filled in or deleted (only if no user facing changes)

See the contribution guide for more details.

Double check this list of stuff that's easy to miss:

Reviewer Notes

If API changes are included, additive changes must be approved by at least two OWNERS and backwards incompatible changes must be approved by more than 50% of the OWNERS, and they must first be added in a backwards compatible way.

Release Notes

Allow Custom Task references in PipelineTask references, which result in Run resources being created.

@tekton-robot tekton-robot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Aug 19, 2020
@tekton-robot tekton-robot requested review from a user and vdemeester August 19, 2020 17:27
@tekton-robot tekton-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Aug 19, 2020
@tekton-robot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign imjasonh
You can assign the PR to them by writing /assign @imjasonh in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@imjasonh
Copy link
Member Author

/kind feature

@tekton-robot tekton-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Aug 19, 2020
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1beta1/pipeline_validation.go 96.1% 96.2% 0.1
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go 90.2% 86.6% -3.6

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1beta1/pipeline_validation.go 96.1% 96.2% 0.1
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go 90.2% 86.6% -3.6

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1beta1/pipeline_validation.go 96.1% 96.2% 0.1

@tekton-robot tekton-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 21, 2020
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1beta1/pipeline_validation.go 96.1% 96.2% 0.1
pkg/reconciler/pipelinerun/pipelinerun.go 86.3% 82.2% -4.1
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go 90.2% 86.6% -3.5

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1beta1/pipeline_validation.go 96.1% 96.2% 0.1
pkg/reconciler/pipelinerun/pipelinerun.go 86.3% 83.1% -3.2
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go 90.2% 86.7% -3.5

@tekton-robot tekton-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 24, 2020
@tekton-robot tekton-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 25, 2020
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1beta1/pipeline_validation.go 96.1% 96.2% 0.1
pkg/reconciler/pipelinerun/pipelinerun.go 86.3% 83.1% -3.2
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go 90.2% 85.5% -4.7

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1beta1/pipeline_validation.go 96.1% 96.2% 0.1
pkg/reconciler/pipelinerun/pipelinerun.go 86.3% 83.1% -3.2
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go 90.2% 85.5% -4.7

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1beta1/pipeline_validation.go 96.1% 96.2% 0.1
pkg/reconciler/pipelinerun/pipelinerun.go 86.3% 83.1% -3.2
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go 90.2% 85.4% -4.7

@imjasonh imjasonh added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 25, 2020
@tekton-robot tekton-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 25, 2020
@tektoncd tektoncd deleted a comment from tekton-robot Aug 25, 2020
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1beta1/pipeline_validation.go 96.1% 96.2% 0.1
pkg/reconciler/pipelinerun/pipelinerun.go 86.3% 86.4% 0.1
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go 90.2% 87.8% -2.3

@imjasonh
Copy link
Member Author

/test pull-tekton-pipeline-integration-tests

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1beta1/pipeline_validation.go 96.1% 96.2% 0.1
pkg/reconciler/pipelinerun/pipelinerun.go 86.3% 86.4% 0.1
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go 90.2% 87.8% -2.3

@GregDritschler
Copy link
Contributor

GregDritschler commented Aug 27, 2020

There are a couple of other TaskRun-related functions in pipelinerun.go that might need equivalents for Run. I think these are for corner case situations so it's not needed immediately but maybe just put TODOs to revisit them.

updateTaskRunsStatusDirectly: This is called on the pr.isDone() path to re-sync all of the TaskRun statuses. I'm not exactly sure why that's necessary; I would have thought they all had to be synced to make a decision the PR is done and can't change after that. But I guess there may be corner cases where that's not true.

updatePipelineRunStatusFromInformer: This looks for existing TaskRuns associated with the PR and makes sure the PR has their statuses. It's intended to cover a window where the reconciler creates a TR but then fails when updating the PR status. Without this, the next reconcile would create another TR because it lost track of it. Similar logic is needed for Runs but it's definitely a corner case.

TaskRun *v1beta1.TaskRun

// If the PipelineTask is a Custom Task, RunName and Run will be set.
RunName string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is more of an FYI than a comment but I noticed that our changes overlap here quite a bit. Overall, I think our approach to resolving pipeline tasks could use a cleaning up since now its OCI images and CustomTasks but who knows what else we will need to resolve in the future.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, absolutely! I was mostly trying to change as little as possible with this change, and try to keep track of cleanups for future PRs. If you have things in mind could you open an issue to help track them? I can add some smells I've noticed too.

@imjasonh imjasonh mentioned this pull request Aug 28, 2020
4 tasks
@imjasonh
Copy link
Member Author

imjasonh commented Sep 1, 2020

There are a couple of other TaskRun-related functions in pipelinerun.go that might need equivalents for Run. I think these are for corner case situations so it's not needed immediately but maybe just put TODOs to revisit them.

updateTaskRunsStatusDirectly: This is called on the pr.isDone() path to re-sync all of the TaskRun statuses. I'm not exactly sure why that's necessary; I would have thought they all had to be synced to make a decision the PR is done and can't change after that. But I guess there may be corner cases where that's not true.

Yeah I agree, AIUI in theory this shouldn't be necessary -- it's updating TaskRun statuses to match the PipelineRun's view, which shouldn't be necessary.

It's also kind of gross that TaskRuns are given names before they're ever run, so errors.IsNotFound is treated as expected (indicating not-run-yet) instead of a real error. We might want to take this opportunity to rethink this behavior.

updatePipelineRunStatusFromInformer: This looks for existing TaskRuns associated with the PR and makes sure the PR has their statuses. It's intended to cover a window where the reconciler creates a TR but then fails when updating the PR status. Without this, the next reconcile would create another TR because it lost track of it. Similar logic is needed for Runs but it's definitely a corner case.

This also seems like a smell to me, we're supposed to be enqueueing PipelineRun reconciliations when TaskRun statuses change already, as well as doing periodic reconciliations in case we miss something. (This was introduced in #2573 for context)

I'll add a TODO in both cases, but before I do them I'll make sure it's actually necessary at all.

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1beta1/pipeline_validation.go 96.1% 96.3% 0.3
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go 90.2% 87.8% -2.3

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1beta1/pipeline_validation.go 96.1% 96.3% 0.3
pkg/reconciler/pipelinerun/pipelinerun.go 86.3% 86.6% 0.3
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go 90.2% 87.8% -2.3

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/apis/pipeline/v1beta1/pipeline_validation.go 96.1% 96.3% 0.3
pkg/reconciler/pipelinerun/pipelinerun.go 86.3% 86.6% 0.3
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go 90.2% 87.8% -2.3

@tekton-robot
Copy link
Collaborator

@imjasonh: The following test failed, say /retest to rerun them all:

Test name Commit Details Rerun command
pull-tekton-pipeline-integration-tests 0b4f7ee link /test pull-tekton-pipeline-integration-tests

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

1 similar comment
@tekton-robot
Copy link
Collaborator

@imjasonh: The following test failed, say /retest to rerun them all:

Test name Commit Details Rerun command
pull-tekton-pipeline-integration-tests 0b4f7ee link /test pull-tekton-pipeline-integration-tests

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@GregDritschler
Copy link
Contributor

updatePipelineRunStatusFromInformer: This looks for existing TaskRuns associated with the PR and makes sure the PR has their statuses. It's intended to cover a window where the reconciler creates a TR but then fails when updating the PR status. Without this, the next reconcile would create another TR because it lost track of it. Similar logic is needed for Runs but it's definitely a corner case.

This also seems like a smell to me, we're supposed to be enqueueing PipelineRun reconciliations when TaskRun statuses change already, as well as doing periodic reconciliations in case we miss something. (This was introduced in #2573 for context)

updatePipelineRunStatusFromInformer does cover a potential exposure. The pipelinerun reconciler might fail to update the PR status after having created a TR. On the next reconcile it won't have any record of having created a TR so it will create it again, resulting in two TRs for the same pipeline task. There is an open real-life report of this happening: #3126. It's happening despite this function being there, so either there's a hole in it or there's something else going on.

Copy link
Member

@vdemeester vdemeester left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think extracting the Run part of the go api to it's own package (like resources) will ease the import cycle & co pain.

Few comments but overall looks good.

// PipelineTaskName is the name of the PipelineTask.
PipelineTaskName string `json:"pipelineTaskName,omitempty"`

// TODO(#3133): Add v1alpha1.RunStatus here, without introducing an import cycle.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it a TODO for this PR or a follow-up ?
One way would be to do like pipeline resources, aka extracting the run v1alpha1 go api in its own package.

Comment on lines +211 to +212
err := c.reconcile(ctx, pr)
if err != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason for not doing if err := c.reconcile(ctx, pr); err != nil { instead ?

if t.TaskRun == nil {
return false
}
return t.TaskRun.IsSuccessful()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😻

Comment on lines +252 to +255
if _, ok := candidateTasks[t.PipelineTask.Name]; ok && t.TaskRun == nil && t.Run == nil {
tasks = append(tasks, t)
}
if _, ok := candidateTasks[t.PipelineTask.Name]; ok && t.TaskRun != nil {
status := t.TaskRun.Status.GetCondition(apis.ConditionSucceeded)
if _, ok := candidateTasks[t.PipelineTask.Name]; ok {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find the following easier to follow that current code.

if _, ok := candidateTasks[t.PipelineTask.Name]; ok {
	if t.TaskRun == nil && t.Run == nil {
		tasks = append(tasks, t)
	}
	// […]
}

or at least, extracting _, ok := candidateTasks[t.PipelineTask.Name] into its own variable and using it in the if or a switch/case 🙃

_ "k8s.io/client-go/plugin/pkg/client/auth/oidc"
knativetest "knative.dev/pkg/test"
"knative.dev/pkg/test/logging" // Mysteriously by k8s libs, or they fail to create `KubeClient`s from config. Apparently just importing it is enough. @_@ side effects @_@. https://github.com/kubernetes/client-go/issues/242
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment seems misplaced — same goes for the one above.. I wonder how it passes the linter 🤔

@tekton-robot tekton-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 8, 2020
@tekton-robot
Copy link
Collaborator

@imjasonh: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@imjasonh
Copy link
Member Author

Closing in favor of #3463

@imjasonh imjasonh closed this Nov 12, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Categorizes issue or PR as related to a new feature. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants