Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix the Tekton controller panic for Metrics #5166

Merged
merged 1 commit into from
Aug 9, 2022
Merged

Fix the Tekton controller panic for Metrics #5166

merged 1 commit into from
Aug 9, 2022

Conversation

khrm
Copy link
Contributor

@khrm khrm commented Jul 19, 2022 •
edited by vdemeester
Loading

If taskrun/pipelinerun completion is watch by another program
and deleted as soon as it is done, then metrics won't be able to
fetch it and should return.
Fixes #5165

Changes

Submitter Checklist

As the author of this PR, please check off the items in this checklist:

  • Has Docs included if any changes are user facing
  • Has Tests included if any functionality added or changed
  • Follows the commit message standard
  • Meets the Tekton contributor standards (including
    functionality, content, code)
  • Has a kind label. You can add one by adding a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
  • Release notes block below has been updated with any user facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings)
  • Release notes contains the string "action required" if the change requires additional action from users switching to the new release

Release Notes

Fix the Tekton controller panic for Metrics.

@tekton-robot tekton-robot added the do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. label Jul 19, 2022
@tekton-robot tekton-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Jul 19, 2022
@khrm khrm changed the title Fix the Tekton controller panic for Metrics WIP: Fix the Tekton controller panic for Metrics Jul 19, 2022
@tekton-robot tekton-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 19, 2022
@khrm
Copy link
Contributor Author

khrm commented Jul 19, 2022

Trying to test this fix.

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/pipelinerun.go 85.8% 85.7% -0.1
pkg/reconciler/taskrun/taskrun.go 80.3% 80.1% -0.2

@khrm
Copy link
Contributor Author

khrm commented Jul 19, 2022

/kind bug

@tekton-robot tekton-robot added the kind/bug Categorizes issue or PR as related to a bug. label Jul 19, 2022
@khrm khrm changed the title WIP: Fix the Tekton controller panic for Metrics Fix the Tekton controller panic for Metrics Jul 26, 2022
@tekton-robot tekton-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Jul 26, 2022
@tekton-robot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vdemeester

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tekton-robot tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 26, 2022
@dibyom
Copy link
Member

dibyom commented Jul 26, 2022

Thanks @khrm Do we have any tests for the metrics recording which might help catch this sort of bug early?

@tekton-robot tekton-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Aug 5, 2022
khrm
khrm commented Aug 5, 2022
View reviewed changes
Copy link
Contributor Author

@khrm khrm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dibyom We don't have any such tests for metrics. Or even e2e tests. We do have an outstanding issue regarding this.

@khrm
Fix the Tekton controller panic for Metrics
26e452b 
If taskrun/pipelinerun completion is watch by another program
and deleted as soon as it is done, then metrics won't be able to
fetch it and should return.
khrm
khrm commented Aug 5, 2022
View reviewed changes
Copy link
Contributor Author

@khrm khrm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/retest

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/pipelinerun.go 85.8% 85.6% -0.3
pkg/reconciler/taskrun/taskrun.go 80.6% 80.3% -0.4

khrm
khrm commented Aug 9, 2022
View reviewed changes
Copy link
Contributor Author

@khrm khrm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@abayer Can we get this merged? @afrittoli @dibyom

@abayer
Copy link
Contributor

abayer commented Aug 9, 2022

/lgtm

@tekton-robot tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Aug 9, 2022
@tekton-robot tekton-robot merged commit 55f0441 into tektoncd:main Aug 9, 2022
Epsilon314
Epsilon314 reviewed Aug 29, 2022
View reviewed changes
pkg/reconciler/pipelinerun/pipelinerun.go
logger.Errorf("Error getting PipelineRun %s when updating metrics: %w", pr.Name, err)
return
} else if kerrors.IsNotFound(err) {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If err IsNotFound, newPr is nil, thus L257 panic.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Will you raise a new pr?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see you didn't raise a pr, I am creating one for taskrun/pipelinerun referencing you.

@khrm khrm deleted the b/5165 branch August 29, 2022 08:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@abayer abayer abayer left review comments

@Epsilon314 Epsilon314 Epsilon314 left review comments

@vdemeester vdemeester vdemeester approved these changes

@dibyom dibyom Awaiting requested review from dibyom

@pritidesai pritidesai Awaiting requested review from pritidesai

Assignees

@abayer abayer

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. kind/bug Categorizes issue or PR as related to a bug. lgtm Indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
6 participants
@khrm @tekton-robot @dibyom @abayer @vdemeester @Epsilon314