Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restarts KES while deploying prow #26493

Merged
merged 1 commit into from
Jun 6, 2022

Conversation

chaodaiG
Copy link
Contributor

@chaodaiG chaodaiG commented Jun 6, 2022

Prow now authenticates with build clusters with tokens that are valid for 2 days. The token is refreshed by a prow job https://prow.k8s.io/?type=periodic&job=ci-test-infra-gencred-refresh-kubeconfig and stores in GCP secret manager, KES is responsible for syncing the secrets into prow. Have observed KES being flaky at time to time, generally more than 10 days after the KES pods started running. See #24869 (comment)

This is a temporary solution aim to mitigate the issue of long running KES pods

Prow now authenticates with build clusters with tokens that are valid for 2 days. The token is refreshed by a prow job https://prow.k8s.io/?type=periodic&job=ci-test-infra-gencred-refresh-kubeconfig and stores in GCP secret manager, KES is responsible for syncing the secrets into prow. Have observed KES being flaky at time to time, generally more than 10 days after the KES pods started running. See kubernetes#24869 (comment)

This is a temporary solution aim to mitigate the issue of long running KES pods
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. area/config Issues or PRs related to code in /config sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Jun 6, 2022
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chaodaiG

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 6, 2022
@ameukam
Copy link
Member

ameukam commented Jun 6, 2022

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 6, 2022
@k8s-ci-robot k8s-ci-robot merged commit 9395ed7 into kubernetes:master Jun 6, 2022
@k8s-ci-robot k8s-ci-robot added this to the v1.25 milestone Jun 6, 2022
@chaodaiG chaodaiG deleted the restart-kes-periodically branch July 11, 2022 21:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/config Issues or PRs related to code in /config cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants