Startup issue with KIC wipes database

[ichbinfrog]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

In a certain startup scenario, KIC nukes all pre-existing config in the Kong Admin API that's not managed by it due to how tag filtering gets configured.

Possibly related to #2274

Expected Behavior

Tag filtering should be on by default to avoid removing pre-existing config in case of failure.

Steps To Reproduce

1. In a Kubernetes cluster, with:

- Kong Ingress Controller (3.1) deployed via helm (2.34.0) without any FeatureGates enabled
- Kong Gateway (3.6) deployed via helm (2.34.0)
- Those two are deployed in separate helm releases and not via the kong ingress helm chart
- Gateway API deployed (https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.0.0/experimental-install.yaml)


2. Create Kong Consumer / other Kong resources that are do not have the `managed-by-ingress-controller` tag via the Admin API

3. Scale down the gateway API to 0 replicas. 

4. Rerollout the Ingress Controller and observe the logs of the elected leader until the following log appears: `tag filtering disabled because Kong Admin API does not support tags`

5. Rescale up the Kong Gateway. 

6. Observe as it wipes non Ingress Controller managed configs and removes the `managed-by-ingress-controller` tag from all currently managed objects

Kong Ingress Controller version

Kong Ingress Controller (3.1) 
Kong Gateway (3.6)

Kubernetes version

Independent of k8s version

Anything else?

No response

[ichbinfrog]

This issue constantly occurs on our clusters making the Kong Ingress Controller not viable for production. Some further clarifications:

• It happens despite explicitly disabling the KongConsumer controller via environment variables

CONTROLLER_ENABLE_CONTROLLER_KONGCONSUMER: false

• Sometimes no log occurs and the Ingress Controller starts removing all tags from managed

updating service redacted  {
  "tags": [
  - "managed-by-ingress-controller"
  ]
}

deleting consumers redacted

I think this is due to the error handling done in:

kubernetes-ingress-controller/internal/dataplane/sendconfig/kong.go

Lines 64 to 69 in 9783552

	ok, err := cl.AdminAPIClient().Tags.Exists(ctx)
	if err != nil {
	return fmt.Errorf("Kong Admin API (%s) does not support tags: %w", cl.BaseRootURL(), err)
	}
	if !ok {
	return fmt.Errorf("Kong Admin API (%s) does not support tags", cl.BaseRootURL())

Shouldn't the default behavior be that tags are considered as enabled as long as this config is set?

kubernetes-ingress-controller/internal/dataplane/sendconfig/kong.go

Line 28 in 9783552

FilterTags []string