KongConfigurationTranslationFailed

[balait4]

Hi, After upgrade from 3.5.0 to 3.7.1 and KIC 3.0 to 3.2.3, I'm seeing the below issue.
This is dbless and I upgraded manually, I applied the roles and crd from the below link.
But still its not working, seems something missing in roles, could thought/help on this please?

kubectl kustomize https://github.com/Kong/kubernetes-ingress-controller/config/crd?ref=v3.2.3 | kubectl apply -f -
kubectl kustomize https://github.com/Kong/kubernetes-ingress-controller/config/rbac?ref=v3.2.3 | kubectl apply -f -

I also installed the Install the Gateway APIs as per the https://docs.konghq.com/kubernetes-ingress-controller/latest/install/helm/#installation for first time as we dont have that before.

 2024-09-06T13:13:35Z error recording a Warning event for object {"name": "northamerica-lb-header", "namespace": "api-proxy", "kind": "Service", "apiVersion": "/v1", "reason": "KongConfigurationTranslationFailed", "message": "failed fetching KongUpstreamPolicy: KongUpstreamPolicy api-proxy/kongingress-lb-header not found", "error": "object failed to apply"}
 2024-09-06T13:13:35Z error recording a Warning event for object {"name": "britishisles-least-conn", "namespace": "api-proxy", "kind": "Service", "apiVersion": "/v1", "reason": "KongConfigurationTranslationFailed", "message": "failed fetching KongUpstreamPolicy: KongUpstreamPolicy api-proxy/kongingress-least-conn not found", "error": "object failed to apply"}
 2024-09-06T13:13:35Z error recording a Warning event for object {"name": "netherlands-funda-lb-header", "namespace": "api-proxy", "kind": "Service", "apiVersion": "/v1", "reason": "KongConfigurationTranslationFailed", "message": "failed fetching KongUpstreamPolicy: KongUpstreamPolicy api-proxy/kongingress-lb-header not found", "error": "object failed to apply"}
 2024-09-06T13:13:35Z error recording a Warning event for object {"name": "westcentraleurope-lb-header", "namespace": "api-proxy", "kind": "Service", "apiVersion": "/v1", "reason": "KongConfigurationTranslationFailed", "message": "failed fetching KongUpstreamPolicy: KongUpstreamPolicy api-proxy/kongingress-lb-header not found", "error": "object failed to apply"}

[balait4]

@pmalek Any help on this please? I tried applying all the roles and rbac and crds from the KIC v3.2.3, still not working. I tried uninstalling the newly setup Gateway APIs still no luck. But it looks like some roles are missing. Thanks

[balait4]

Is there any detailed log or debug to see why KIC not picking the KongUpstreamPolicy?

[pmalek]

You can try inspecting the KongUpstreamPolicy object status. Its conditions might indicate what's wrong. Additionally you might enable debug logs and search for anything relevant for this KongUpstreamPolicy.

Feel free to share the object status and logs here.

[balait4]

thanks for the reply! Here is the log of KongUpstreamPolicy object status. And KIC debug logs.
KUP_Object_status.txt
debug_log.txt

[balait4]

Hi, One more thing, few KUP is working fine and seems only few are failing but there is no difference in the yaml, here is the working and non-working KUP object.
KUP_non_working_object.txt
KUP_working_object_status.txt

Non-working KUP resource status block is continuously updated with ancestor reordering.

[balait4]

Also attaching the debug log from kong and KIC, I have issue only for the KUP from the namespace api-proxy. KUP from other namespace seems to be working.
debug_log_KIC.txt
debug_log_kong.txt

[balait4]

Any help on this please? As I'm not able to move from version v3.0.2. I want to upgrade to latest version to use the env variable CONTROLLER_GATEWAY_TO_RECONCILE.