From 5224fa5bb7abb1b25125682a4e363be990ed2aff Mon Sep 17 00:00:00 2001 From: Chao Dai Date: Mon, 6 Jun 2022 08:26:18 -0700 Subject: [PATCH] Restarts KES while deploying prow Prow now authenticates with build clusters with tokens that are valid for 2 days. The token is refreshed by a prow job https://prow.k8s.io/?type=periodic&job=ci-test-infra-gencred-refresh-kubeconfig and stores in GCP secret manager, KES is responsible for syncing the secrets into prow. Have observed KES being flaky at time to time, generally more than 10 days after the KES pods started running. See https://github.com/kubernetes/test-infra/issues/24869#issuecomment-1147530320 This is a temporary solution aim to mitigate the issue of long running KES pods --- config/prow/Makefile | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/config/prow/Makefile b/config/prow/Makefile index 1e68c5436e3d..9ed06cd8bfd6 100644 --- a/config/prow/Makefile +++ b/config/prow/Makefile @@ -35,6 +35,11 @@ deploy-prow: get-cluster-credentials kubectl apply --server-side=true -f ./cluster/prowjob-crd/prowjob_customresourcedefinition.yaml kubectl apply -f ./cluster/ + # Temporary solution for working around the issue of KES being flaky, see + # https://github.com/kubernetes/test-infra/issues/24869#issuecomment-1147530320. + # TODO(chaodaiG): remove this once the above issue is fixed. + kubectl rollout restart deployment kubernetes-external-secrets + deploy-build: get-build-cluster-credentials kubectl apply -f ./cluster/build/