apikey is not updated within kong on kongCredential update

[dcherniv]

Summary

When kongCredential is updated and applied the changes are not reflected in kong.
example credential created:

apiVersion: configuration.konghq.com/v1
kind: KongCredential
metadata:
  name: int-rob
consumerRef: int-rob
type: key-auth
config:
  key: Tlp0VTVNazl7dnhuIjksaAo

Which then was updated to and applied through helm upgrade:

apiVersion: configuration.konghq.com/v1
kind: KongCredential
metadata:
  name: int-rob
consumerRef: int-rob
type: key-auth
config:
  key: Tlp0VTVNazl7dnhuIjksaAo=

[root@intminion01 templates]# kubectl get kongcredentials -n int  -o yaml int-rob
apiVersion: configuration.konghq.com/v1
config:
  key: Tlp0VTVNazl7dnhuIjksaAo=
consumerRef: int-rob
kind: KongCredential
metadata:
  clusterName: ""
  creationTimestamp: "2019-01-27T03:26:40Z"
  generation: 1
  name: int-rob
  namespace: int
  resourceVersion: "109324196"
  selfLink: /apis/configuration.konghq.com/v1/namespaces/int/kongcredentials/int-rob
  uid: 5cc6c767-21e3-11e9-a3db-42010a8e0177
type: key-auth
[root@intminion01 templates]# 

The change is not reflected:

/ # curl http://localhost:8001/consumers/int-rob/key-auth
{"next":null,"data":[{"key":"Tlp0VTVNazl7dnhuIjksaAo","created_at":1548559924,"consumer":{"id":"5ccf25c9-21e3-11e9-a3db-42010a8e0177"},"id":"5cc6c767-21e3-11e9-a3db-42010a8e0177"}]}/

Kong Ingress controller version
0.3.0

Kong or Kong Enterprise version
kong community 1.0.2

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.2", GitCommit:"cff46ab41ff0bb44d8584413b598ad8360ec1def", GitTreeState:"clean", BuildDate:"2019-01-10T23:35:51Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.11-gke.1", GitCommit:"5c4fddf874319c9825581cc9ab1d0f0cf51e1dc9", GitTreeState:"clean", BuildDate:"2018-11-30T16:18:58Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

Environment

• Cloud provider or hardware configuration: gke
• OS (e.g. from /etc/os-release): cOS
• Kernel (e.g. uname -a): Linux kong-kong-ingress-6f54f64ff6-dntvb 4.14.65+ #1 SMP Thu Oct 25 10:42:50 PDT 2018 x86_64 Linux
• Install tools: helm
• Others:

What happened

kongCredential was updated and applied through helm upgrade

Expected behavior

the new apikey applied to kong

Steps To Reproduce

1. Create kongCredential resource type with apikey set some value
2. Update the value.
3. Run helm upgrade
4. The key is not updated

[dcherniv]

Some more info,
It appears its not just apikey. At some point, past 8 consumers, there seems to be some issue with associating credentials to consumers.
Sometimes acl group membership would not populate, sometimes apikey wouldn't.
It's really puzzling because all of the kubernetes resources are generated automatically, exactly the same though a ci/cd pipeline.

It really looks like some kind of race condition.

Perhaps helm creates resources in a certain order? does consumer have to exist before the credentials are created with consumerRef?

UPDATE: it also appears if i go ahead and rename the credential by adding 2 to the end, it does get created, whereas any manipulation on the credential without changing the name has no effect.

apiVersion: configuration.konghq.com/v1
config:
  key: Oz9nUkNHNnQ3ZUVkKTZ0TlFUTS8K
consumerRef: dev-merchandise
kind: KongCredential
metadata:
  name: dev-merchandise2
  namespace: dev
type: key-auth

Full sample:

apiVersion: v1
items:
- apiVersion: configuration.konghq.com/v1
  kind: KongConsumer
  metadata:
    name: dev-merchandise
  username: dev-merchandise
kind: List
---
apiVersion: configuration.konghq.com/v1
kind: KongCredential
metadata:
  name: dev-merchandise
consumerRef: dev-merchandise
type: key-auth
config:
  key: Oz9nUkNHNnQ3ZUVkKTZ0TlFUTS8K
---
apiVersion: configuration.konghq.com/v1
kind: KongCredential
metadata:
  name: dev-merchandise-group
consumerRef: dev-merchandise
type: acl
config:
  group: "dev-merchandise-group"

@hbagdi

[Kunone]

Exactly same issue we are facing. And it doesn't sync to KONG for DELETE as well. I had a quick look at the code kong.go line 656 which clearly not supporting changes (update or delete) to credential itself. Seems we have to delete consumer in order to update key.

[hbagdi]

The following is currently not a bug but an expected behavior which is not documented:

Once a credential is created in Kong using KongCredential CRD, it won't be updated.

This is a bit of a challenge because not all credentials can be diffed, for example, basic-auth passwords in Kong are hashed, making it impossible to figure out if a credential is same as the one in KongCredential resource.

We've not yet figured out a path forward for this problem yet.

[hbagdi]

This is implemented with #371 and will be part of 0.6.0 release.