Second controller with controller-class configured listens to incorrect IngressClass

[MarkMckessock]

NGINX Ingress controller version: v1.0.0

Kubernetes version (use kubectl version): v1.20.7+k3s1

Environment: Bare-metal

• Cloud provider or hardware configuration: 3x RPi 4GB + R610
• OS (e.g. from /etc/os-release): k3OS v0.20.7-k3s1r0
• Kernel (e.g. uname -a): 5.10.52-v8+ (aarch64)
• Install tools:
  • k3s
• Basic cluster related info:

NAME       STATUS   ROLES                       AGE     VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION   CONTAINER-RUNTIME
hyperion   Ready    control-plane,etcd,master   5d9h    v1.20.7+k3s1   10.0.70.5     <none>        k3OS v0.20.7-k3s1r0            5.10.52-v8+      containerd://1.4.4-k3s1
phoebe     Ready    control-plane,etcd,master   5d9h    v1.20.7+k3s1   10.0.70.4     <none>        k3OS v0.20.7-k3s1r0            5.10.52-v8+      containerd://1.4.4-k3s1
pve01      Ready    worker                      4d21h   v1.21.3+k3s1   10.0.70.2     <none>        Debian GNU/Linux 10 (buster)   5.4.106-1-pve    containerd://1.4.8-k3s1
titan      Ready    control-plane,etcd,master   5d9h    v1.20.7+k3s1   10.0.70.3     <none>        k3OS v0.20.7-k3s1r0            5.10.52-v8+      containerd://1.4.4-k3s1

• How was the ingress-nginx-controller installed:

  • Controller deployed via helm charts using Flux

  • Helm Values'

    • 'nginx' values

    controller:
      ingressClassResource:
        enabled: true
        default: true
      metrics:
        enabled: true
        service:
          annotations:
            prometheus.io/scrape: "true"
            prometheus.io/port: "10254"
          serviceMonitor:
            enabled: true
            namespace: monitoring

    • 'nginx-public' values

    controller:
      ingressClassResource:
        enabled: true
        name: nginx-public
        default: false
        controllerValue: "k8s.io/ingress-nginx-public"
      service:
        loadBalancerIP: ${INGRESS_NGINX_PUBLIC_IP}
      metrics:
        enabled: true
        service:
          annotations:
            prometheus.io/scrape: "true"
            prometheus.io/port: "10254"
          serviceMonitor:
            enabled: true
            namespace: monitoring

• Current State of the controller:

`nginx` Pod Description

Name:         ingress-nginx-controller-84577d6675-9xtkn
Namespace:    ingress-nginx
Priority:     0
Node:         phoebe/10.0.70.4
Start Time:   Tue, 24 Aug 2021 13:48:04 -0400
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/name=ingress-nginx
              pod-template-hash=84577d6675
Annotations:  <none>
Status:       Running
IP:           10.42.2.52
IPs:
  IP:           10.42.2.52
Controlled By:  ReplicaSet/ingress-nginx-controller-84577d6675
Containers:
  controller:
    Container ID:  containerd://e131d2ce6acf4f024b9e8e7f9714a4512ff70b4d50017063bb8e1d73db84d69a
    Image:         k8s.gcr.io/ingress-nginx/controller:v1.0.0@sha256:0851b34f69f69352bf168e6ccf30e1e20714a264ab1ecd1933e4d8c0fc3215c6
    Image ID:      k8s.gcr.io/ingress-nginx/controller@sha256:0851b34f69f69352bf168e6ccf30e1e20714a264ab1ecd1933e4d8c0fc3215c6
    Ports:         80/TCP, 443/TCP, 10254/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      --election-id=ingress-controller-leader
      --controller-class=k8s.io/ingress-nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Running
      Started:      Tue, 24 Aug 2021 13:48:07 -0400
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-84577d6675-9xtkn (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from ingress-nginx-token-sgk4w (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  ingress-nginx-token-sgk4w:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-token-sgk4w
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  kubernetes.io/os=linux
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:          <none>

`nginx-public` Pod Description

Name:         ingress-nginx-public-controller-7bccd8d57-wrnmt
Namespace:    ingress-nginx-public
Priority:     0
Node:         phoebe/10.0.70.4
Start Time:   Tue, 24 Aug 2021 21:56:13 -0400
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx-public
              app.kubernetes.io/name=ingress-nginx
              pod-template-hash=7bccd8d57
Annotations:  <none>
Status:       Running
IP:           10.42.2.61
IPs:
  IP:           10.42.2.61
Controlled By:  ReplicaSet/ingress-nginx-public-controller-7bccd8d57
Containers:
  controller:
    Container ID:  containerd://d9c1fdf2279261e05bbce7289a136f8ef13c94ed49f19ac9d731abbd55c72f08
    Image:         k8s.gcr.io/ingress-nginx/controller:v1.0.0@sha256:0851b34f69f69352bf168e6ccf30e1e20714a264ab1ecd1933e4d8c0fc3215c6
    Image ID:      k8s.gcr.io/ingress-nginx/controller@sha256:0851b34f69f69352bf168e6ccf30e1e20714a264ab1ecd1933e4d8c0fc3215c6
    Ports:         80/TCP, 443/TCP, 10254/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-public-controller
      --election-id=ingress-controller-leader
      --controller-class=k8s.io/ingress-nginx-public
      --configmap=$(POD_NAMESPACE)/ingress-nginx-public-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Running
      Started:      Tue, 24 Aug 2021 21:56:15 -0400
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-public-controller-7bccd8d57-wrnmt (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx-public (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from ingress-nginx-public-token-pr2vw (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-public-admission
    Optional:    false
  ingress-nginx-public-token-pr2vw:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-public-token-pr2vw
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  kubernetes.io/os=linux
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:          <none>

• Current state of ingress object, if applicable:
  • nginx IngressClass

Name:         nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/version=1.0.0
              helm.sh/chart=ingress-nginx-4.0.1
              helm.toolkit.fluxcd.io/name=ingress-nginx
              helm.toolkit.fluxcd.io/namespace=ingress-nginx
Annotations:  ingressclass.kubernetes.io/is-default-class: true
              meta.helm.sh/release-name: ingress-nginx
              meta.helm.sh/release-namespace: ingress-nginx
Controller:   k8s.io/ingress-nginx
Events:       <none>

• nginx-public IngressClass

Name:         nginx-public
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx-public
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/version=1.0.0
              helm.sh/chart=ingress-nginx-4.0.1
              helm.toolkit.fluxcd.io/name=ingress-nginx-public
              helm.toolkit.fluxcd.io/namespace=ingress-nginx-public
Annotations:  meta.helm.sh/release-name: ingress-nginx-public
              meta.helm.sh/release-namespace: ingress-nginx-public
Controller:   k8s.io/ingress-nginx-public
Events:       <none>

• Problem Ingress

Name:             home-assistant
Namespace:        home-assistant
Address:
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
  hass-cert terminates hass.<domain>
Rules:
  Host                    Path  Backends
  ----                    ----  --------
  hass.<domain>
                          /   home-assistant:8123 (10.42.4.12:8123)
Annotations:              cert-manager.io/cluster-issuer: letsencrypt-production
                          kubernetes.io/ingress.class: nginx-public
                          kubernetes.io/tls-acme: true
                          meta.helm.sh/release-name: home-assistant
                          meta.helm.sh/release-namespace: home-assistant
Events:                   <none>

• Others:
  • Controller Logs
    • kubectl -n ingress-nginx logs ingress-nginx-controller-84577d6675-9xtkn | grep home-assistant ('nginx' controller)
      • These logs are as expected with the controller ignoring the ingress tagged with another class.

  I0825 01:59:51.550088       8 store.go:336] "Ignoring ingress because of error while validating ingress class" ingress="home-assistant/home-assistant" error="ingress class annotation is not equal to the expected by Ingress Controller"
  W0825 02:00:18.514507       8 controller.go:1007] Service "home-assistant/home-assistant" does not have any active Endpoint.
  W0825 02:00:18.515269       8 controller.go:1222] Error getting SSL certificate "home-assistant/hass-cert": local SSL certificate home-assistant/hass-cert was not found. Using default certificate
  I0825 02:00:19.209096       8 main.go:101] "successfully validated configuration, accepting" ingress="home-assistant/home-assistant"
  I0825 02:00:19.762450       8 store.go:361] "Ignoring ingress because of error while validating ingress class" ingress="home-assistant/home-assistant" error="ingress class annotation is not equal to the expected by Ingress Controller"
  

  • kubectl -n ingress-nginx-public logs ingress-nginx-public-controller-7bccd8d57-wrnmt | grep home-assistant ('nginx-public' controller)
    • These logs show an issue because this controller should be attached to the nginx-public ingress class.
    • ingress class annotation is not equal to the expected by Ingress Controller would seem to indicate that it is indeed not listening to the correct ingressclass.

  I0825 01:56:20.761729       8 store.go:361] "Ignoring ingress because of error while validating ingress class" ingress="home-assistant/home-assistant" error="ingress class annotation is not equal to the expected by Ingress Controller"
  I0825 01:59:51.543306       8 store.go:336] "Ignoring ingress because of error while validating ingress class" ingress="home-assistant/home-assistant" error="ingress class annotation is not equal to the expected by Ingress Controller"
  W0825 02:00:18.513491       8 controller.go:1007] Service "home-assistant/home-assistant" does not have any active Endpoint.
  W0825 02:00:18.514261       8 controller.go:1222] Error getting SSL certificate "home-assistant/hass-cert": local SSL certificate home-assistant/hass-cert was not found. Using default certificate
  I0825 02:00:19.245263       8 main.go:101] "successfully validated configuration, accepting" ingress="home-assistant/home-assistant"
  I0825 02:00:19.760666       8 store.go:361] "Ignoring ingress because of error while validating ingress class" ingress="home-assistant/home-assistant" error="ingress class annotation is not equal to the expected by Ingress Controller"
  

What happened:

I have created two instances of ingress-nginx: ingress-nginx and ingress-nginx-public. Each lives in their own namespace and has an associated IngressClass (nginx and nginx-public respectively). For the ingress-nginx deployment, I have modified the controller-class value so that it is unique and matches the associated ingress class.

Ingresses assigned to the default nginx class (or none at all) are served properly by the nginx ingress class, but ingresses assigned to the nginx-public class are not served at all and are not assigned an address. From the controller logs, these ingresses are being ignore because the controller is not using the correct ingress class name. It appears that the controller is still listening to the standard nginx ingress class because I can see the loadbalancer IPs for my ingresses bounce back and forth between the addresses assigned to each controller.

What you expected to happen:

The second controller should be properly listening to the nginx-public ingress class and not attempt to server ingresses using the nginx class.

How to reproduce it:

1. Install MetalLB
2. Deploy nginx ingress using helm chart and values specified above
3. Deploy nginx-public ingress using helm chart and values specified above
4. Deploy a service (Home Assistant)

• Deployed using helm chart from https://k8s-at-home.com/charts/
• Chart name: home-assistant
• Relevant ingress config for chart:

  ingress:
    main:
      enabled: true
      annotations:
        kubernetes.io/ingress.class: nginx-public
        kubernetes.io/tls-acme: "true"
        cert-manager.io/cluster-issuer: ${CLUSTER_ISSUER}
      hosts:
        - host: "hass.${DOMAIN}"
          paths:
            - path: /
              pathType: Prefix
      tls:
        - hosts:
            - "hass.${DOMAIN}"
          secretName: hass-cert

5. Ingress is not picked up by controller

/kind bug

[k8s-ci-robot]

@MarkMckessock: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

hi,
Thanks for the detailed information.

Can you kindly check if the known issues like #7510 and #7502 are related .

Also, one other request is, can you kindly edit the data related related to the ingress objects in your report above and actually replace the current text with the output of ;

• kubectl describe ingressclasses -o wide
• kubectl -n <namespace> get ing <ingressname> -o yaml

[longwuyuan]

/assign

[longwuyuan]

Hi, Just as an update, I was able to reproduce the problem and see the error messages in the logs exactly as you described.

Then after reading this, https://kubernetes.io/docs/concepts/services-networking/ingress/#deprecated-annotation ,

• I edited my ingress object configuration.
• I removed the annotation as it is documented as deprecated
• And instead I used the ingressClassName field in the spec of my ingress object

% k explain ingress.spec.ingressClassName
KIND: Ingress
VERSION: networking.k8s.io/v1

FIELD: ingressClassName

DESCRIPTION:
IngressClassName is the name of the IngressClass cluster resource. The
associated IngressClass defines which controller will implement the
resource. This replaces the deprecated kubernetes.io/ingress.class
annotation. For backwards compatibility, when that annotation is set, it
must be given precedence over this field. The controller may emit a warning
if the field and annotation have different values. Implementations of this
API should ignore Ingresses without a class specified. An IngressClass
resource may be marked as default, which can be used to set a default value
for this field. For more information, refer to the IngressClass
documentation.

• That solved the problem for me.

Can you try the ingressClassName field instead of the annotation and update.

[catouc]

I've been having similar issues with v1.0.0 - played with a lot of different configuration that all lead to the same behaviour as described, an ingress reserved for a specific ingressClass which was working under v0.48.1 is now being served by another controller.

Currently I've got a configuration where the documentation for running multiple controllers got our configuration working (with a downtime). This is in conflict with the docs at the user guide multiple ingress section.

Working example config

Controller arguments:

- name: controller
  image: k8s.gcr.io/ingress-nginx/controller:v1.0.0
  imagePullPolicy: IfNotPresent
  lifecycle:
    preStop:
      exec:
        command:
          - /wait-shutdown
  args:
    - /nginx-ingress-controller
    - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
    - --election-id=ingress-controller-leader
    - --ingress-class=k8s.io/default-internal
    - --controller-class=k8s.io/default-internal
    - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
    - --update-status-on-shutdown=false

Matching IngressClass:

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: default-internal
spec:
  controller: k8s.io/default-internal

Ingress using the IngressClass:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: xxx
  namespace: xxx
  labels:
    app: xxx
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 16m
spec:
  ingressClassName: default-internal

Conclusions

• The migration path if you used ingressClasses before v1.0.0 is pretty involved because the spec.controller field is immutable, requiring deletion of the ingressClass - it seems like The new v1.0.0 IngressClass handling logic makes a zero-downtime Ingress controller upgrade hard for the users #7502 is talking about approaches for this in more detail
• The documentation points me in two different directions on how to configure multiple controllers inside of a cluster
  • https://kubernetes.github.io/ingress-nginx/#i-have-more-than-one-controller-running-in-my-cluster-and-i-want-to-use-the-new-spec
  • https://kubernetes.github.io/ingress-nginx/user-guide/multiple-ingress/
• An overview of which flags, --ingress-class or --controller-class are important for what would also help understand the work that is required for a successful migration
  • my assumption is that --ingress-class handles the deprecated annotations and --controller-class handles the ingressClassName and spec.controller values

edit:

We just migrated another system over following this process:

• Delete ingressClass objects
• Recreate new ingressClass objects and start new controllers
• Rolling deployment

It seems like the ingressClass definitions are cached after startup? We did not experience any downtime during this that lead us to this theory - seems a bit counter intuitive but seems to work.
Anyone know more detailed infos on this?

[MarkMckessock]

Thanks @longwuyuan for your quick response. It would seem that you are correct regarding the issue being the deprecated annotations. After adding the spec.ingressClassName field to my ingress, it is now properly being served by the appropriate controller.

@Deichindianer I did not deploy the controller with both the --controller-class and --ingress-class fields, but I would suspect that you are correct regarding the former referencing the v1 resource and the latter handling the deprecated annotation.

[longwuyuan]

@MarkMckessock thanks for confirming. You could close the issue if that settles it.
@Deichindianer thanks a lot for the comments. I have a created a new issue to track the required change to the docs, as you can see.

[longwuyuan]

/kind documentation
/area docs