Skip to content
This repository has been archived by the owner on Feb 22, 2022. It is now read-only.

[stable/kong] Ingress Controller errors when chart is redeployed with Admission Webhook enabled #20050

Closed
krish7919 opened this issue Jan 10, 2020 · 1 comment · Fixed by #20051

Comments

@krish7919
Copy link
Contributor

Describe the bug
I have been using the stable/kong chart for the past few days, and with the admission controller enabled, I see errors like these:

$ kubectl logs -f kong-78f67ff956-rfdlf -c ingress-controller
-------------------------------------------------------------------------------
Kong Ingress controller
  Release:    0.6.2
  Build:      01d61b5
  Repository: [email protected]:kong/kubernetes-ingress-controller.git
  Go:         go1.13.1
-------------------------------------------------------------------------------
I0108 16:28:28.898123       1 main.go:362] Creating API client for https://172.20.0.1:443
I0108 16:28:31.929812       1 main.go:406] Running in Kubernetes Cluster version v1.14+ (v1.14.9-eks-c0eccc) - git (clean) commit c0eccca51d7500bb03b2f163dd8d534ffeb2f7a2 - platform linux/amd64
I0108 16:28:32.066107       1 main.go:148] kong version: 1.4.2
I0108 16:28:32.066130       1 main.go:157] Kong datastore: off
I0108 16:28:32.299271       1 controller.go:242] starting Ingress controller
W0108 16:28:32.299441       1 parser.go:772] service core-services/kong-core-services-proxy does not have any active endpoints
I0108 16:28:32.303670       1 status.go:201] new leader elected: kong-core-services-79465697cf-l7z8l
I0108 16:28:32.457570       1 controller.go:135] successfully synced configuration to Kong
W0108 16:28:35.632898       1 parser.go:772] service core-services/kong-core-services-proxy does not have any active endpoints
I0108 16:28:35.632969       1 controller.go:135] successfully synced configuration to Kong
W0108 16:28:38.966224       1 parser.go:772] service core-services/kong-core-services-proxy does not have any active endpoints
I0108 16:28:38.966335       1 controller.go:135] successfully synced configuration to Kong
W0108 16:28:42.299526       1 parser.go:772] service core-services/kong-core-services-proxy does not have any active endpoints
I0108 16:28:42.299595       1 controller.go:135] successfully synced configuration to Kong
I0108 16:28:48.966263       1 controller.go:135] successfully synced configuration to Kong
I0108 16:29:09.042632       1 status.go:201] new leader elected: kong-core-services-78f67ff956-rfdlf
I0108 16:29:09.042750       1 controller.go:135] successfully synced configuration to Kong
I0108 16:29:36.325703       1 controller.go:135] successfully synced configuration to Kong
I0108 16:30:09.053124       1 status.go:342] updating Ingress core-services/kong-core-services-proxy status to [{ ae659a862323311eaa0b30ab004b4262-fc8c9758442b399a.elb.eu-central-1.amazonaws.com}]
I0108 16:30:09.058899       1 controller.go:135] successfully synced configuration to Kong
I0108 16:32:43.876922       1 controller.go:135] successfully synced configuration to Kong
I0108 16:33:35.205628       1 controller.go:135] successfully synced configuration to Kong
I0108 16:34:19.355810       1 controller.go:135] successfully synced configuration to Kong
I0108 16:34:22.689274       1 controller.go:135] successfully synced configuration to Kong
I0108 16:34:26.022593       1 controller.go:135] successfully synced configuration to Kong
E0108 16:34:32.104437       1 leaderelection.go:324] error retrieving resource lock core-services/kong-ingress-controller-leader-kong-kong: Unauthorized
E0108 16:34:39.610381       1 leaderelection.go:324] error retrieving resource lock core-services/kong-ingress-controller-leader-kong-kong: Unauthorized
E0108 16:34:49.111139       1 reflector.go:283] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to watch *v1.KongCredential: the server has asked fo
r the client to provide credentials (get kongcredentials.configuration.konghq.com)
E0108 16:34:50.117085       1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.KongCredential: Unauthorized
E0108 16:34:51.122745       1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.KongCredential: Unauthorized
E0108 16:34:55.152849       1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.KongCredential: Unauthorized
E0108 16:34:56.163018       1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.KongCredential: Unauthorized
E0108 16:34:57.168777       1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.KongCredential: Unauthorized
E0108 16:34:57.271003       1 reflector.go:283] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to watch *v1.Secret: the server has asked for the cl
ient to provide credentials (get secrets)
E0108 16:34:58.174272       1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.KongCredential: Unauthorized
E0108 16:34:58.276510       1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Secret: Unauthorized
...
...

This happens with ingress controller versions 0.6.2 and 0.7.0

Version of Helm and Kubernetes:
K8s: 1.14
Helm: 2.x

Which chart:
stable/kong

What happened:
I noticed that this happens on a second/repeated deploy of the chart.

What you expected to happen:
No errors when the chart is redeployed repeatedly without any changes.

How to reproduce it (as minimally and precisely as possible):
Redeploy the chart repeatedly with no changes, the ingress-controller will start to fail within a few mins.

Anything else we need to know:
This happens due to this code in this line:

{{- $ca := genCA "kong-admission-ca" 3650 -}}
{{- $cert := genSignedCert $cn nil nil 3650 $ca -}}

This code generates a new set of CA and client certs, which updates the corresponding K8s Secret, but the ingress-controller does not restart to use the new set of certificates.
A checksum check on the Secret should fix the issue.

@krish7919
Copy link
Contributor Author

I will send a PR shortly.

@krish7919 krish7919 changed the title [stable/kong] Admission Webhook errors when charrt is redeployed [stable/kong] Admission Webhook errors when chart is redeployed Jan 10, 2020
@krish7919 krish7919 changed the title [stable/kong] Admission Webhook errors when chart is redeployed [stable/kong] Ingress Controller errors when chart is redeployed with Admission Webhook enabled Jan 10, 2020
rainest pushed a commit to Kong/charts that referenced this issue Feb 4, 2021
On review of the history behind this, we don't need this checksum if we
don't expect the certificate to change. We added it as a workaround for
failures caused by the certificate rotating on update:
helm/charts#20050
rainest pushed a commit to Kong/charts that referenced this issue Feb 6, 2021
On review of the history behind this, we don't need this checksum if we
don't expect the certificate to change. We added it as a workaround for
failures caused by the certificate rotating on update:
helm/charts#20050
rainest pushed a commit to Kong/charts that referenced this issue Feb 9, 2021
On review of the history behind this, we don't need this checksum if we
don't expect the certificate to change. We added it as a workaround for
failures caused by the certificate rotating on update:
helm/charts#20050
rainest pushed a commit to Kong/charts that referenced this issue Mar 4, 2021
On review of the history behind this, we don't need this checksum if we
don't expect the certificate to change. We added it as a workaround for
failures caused by the certificate rotating on update:
helm/charts#20050
ubergesundheit pushed a commit to giantswarm/kong-app that referenced this issue Jun 14, 2021
On review of the history behind this, we don't need this checksum if we
don't expect the certificate to change. We added it as a workaround for
failures caused by the certificate rotating on update:
helm/charts#20050
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant