nginx.ingress.kubernetes.io/configuration-snippet stopped being applied in helm 4.8.0

[StefanLobbenmeierObjego]

What happened:

I upgraded from 4.7.2 to 4.8.0 and expected everything to still work.

After the upgrade a pod no longer got the correct requests. I investigated why and discovered that the nginx.conf is no longer generated correctly:

That is the ingress.yaml of the pod that now gets wrong requests:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: pgadmin-oauth
  namespace: tools
  annotations:
     cert-manager.io/acme-challenge-type: http01
     cert-manager.io/cluster-issuer: letsencrypt
     nginx.ingress.kubernetes.io/ssl-redirect: "true"
     nginx.ingress.kubernetes.io/use-regex: "true"
     nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-Script-Name /pgadmin;

in v 4.7.2 I still get the snippet in my configuration:

proxy_set_header X-Script-Name /pgadmin;

and in v4.8.0 it is now missing:

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

/nginx-ingress-controller --version

NGINX Ingress controller
Release: v1.9.0
Build: 4bd3d6b
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.21.6

---

Kubernetes version (use kubectl version):
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.25.4
WARNING: version difference between client (1.28) and server (1.25) exceeds the supported minor version skew of +/-1

Environment:

• Cloud provider or hardware configuration: Azure

• OS (e.g. from /etc/os-release): Ubuntu 22.04.1 LTS

• Kernel (e.g. uname -a): Linux nginx-ingress-nginx-controller-7f58f9c446-6l2dt 5.15.0-1030-azure Nginx Ingress Controller ConfigMap support for header size  #37-Ubuntu SMP Mon Dec 12 19:15:51 UTC 2022 x86_64 Linux

• Install tools:

  • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.

• Basic cluster related info:

  • kubectl version
    Client Version: v1.28.2
    Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
    Server Version: v1.25.4
    WARNING: version difference between client (1.28) and server (1.25) exceeds the supported minor version skew of +/-1
  • kubectl get nodes -o wide
    NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
    aks-intmain-10577288-vmss000000 Ready agent 238d v1.25.4 10.60.0.116 Ubuntu 22.04.1 LTS 5.15.0-1030-azure containerd://1.6.15+azure-1
    aks-intmain-10577288-vmss000001 Ready agent 238d v1.25.4 10.60.2.39 Ubuntu 22.04.1 LTS 5.15.0-1030-azure containerd://1.6.15+azure-1
    aks-intmain-10577288-vmss000002 Ready agent 238d v1.25.4 10.60.2.148 Ubuntu 22.04.1 LTS 5.15.0-1030-azure containerd://1.6.15+azure-1

• How was the ingress-nginx-controller installed:

  • If helm was used then please show output of helm ls -A | grep -i ingress
    nginx ingress 29 2023-09-27 13:47:27.895964 +0200 CEST deployed ingress-nginx-4.8.0 1.9.0

  • If helm was used then please show output of helm -n <ingresscontrollernamepspace> get values <helmreleasename>
    USER-SUPPLIED VALUES:
    controller:
    metrics:
    enabled: true
    podAnnotations:
    prometheus.io/port: 10254
    prometheus.io/scrape: true
    service:
    annotations:
    service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz
    externalTrafficPolicy: Local

  • If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used

  • if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances

• Current State of the controller:

  • kubectl describe ingressclasses
    Name: nginx
    Labels: app.kubernetes.io/component=controller
    app.kubernetes.io/instance=nginx
    app.kubernetes.io/managed-by=Helm
    app.kubernetes.io/name=ingress-nginx
    app.kubernetes.io/part-of=ingress-nginx
    app.kubernetes.io/version=1.9.0
    helm.sh/chart=ingress-nginx-4.8.0
    Annotations: meta.helm.sh/release-name: nginx
    meta.helm.sh/release-namespace: ingress
    Controller: k8s.io/ingress-nginx
    Events:

  • kubectl -n <ingresscontrollernamespace> get all -A -o wide
    that is a huge amount of text I would like to avoid

  • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
    Name: nginx-ingress-nginx-controller-7995ddbbc4-fgh5g
    Namespace: ingress
    Priority: 0
    Service Account: nginx-ingress-nginx
    Node: aks-intmain-10577288-vmss000001/10.60.2.39
    Start Time: Wed, 27 Sep 2023 13:47:39 +0200
    Labels: app.kubernetes.io/component=controller
    app.kubernetes.io/instance=nginx
    app.kubernetes.io/managed-by=Helm
    app.kubernetes.io/name=ingress-nginx
    app.kubernetes.io/part-of=ingress-nginx
    app.kubernetes.io/version=1.9.0
    helm.sh/chart=ingress-nginx-4.8.0
    pod-template-hash=7995ddbbc4
    Annotations: prometheus.io/port: 10254
    prometheus.io/scrape: true
    Status: Running
    IP: 10.60.2.109
    IPs:
    IP: 10.60.2.109
    Controlled By: ReplicaSet/nginx-ingress-nginx-controller-7995ddbbc4
    Containers:
    controller:
    Container ID: containerd://d8b9d655b431edc48a4c185891d6ec784f57df0837f2c318678416bf2130bc7c
    Image: registry.k8s.io/ingress-nginx/controller:v1.9.0@sha256:c15d1a617858d90fb8f8a2dd60b0676f2bb85c54e3ed11511794b86ec30c8c60
    Image ID: registry.k8s.io/ingress-nginx/controller@sha256:c15d1a617858d90fb8f8a2dd60b0676f2bb85c54e3ed11511794b86ec30c8c60
    Ports: 80/TCP, 443/TCP, 10254/TCP, 8443/TCP
    Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
    /nginx-ingress-controller
    --publish-service=$(POD_NAMESPACE)/nginx-ingress-nginx-controller
    --election-id=nginx-ingress-nginx-leader
    --controller-class=k8s.io/ingress-nginx
    --ingress-class=nginx
    --configmap=$(POD_NAMESPACE)/nginx-ingress-nginx-controller
    --validating-webhook=:8443
    --validating-webhook-certificate=/usr/local/certificates/cert
    --validating-webhook-key=/usr/local/certificates/key
    State: Running
    Started: Wed, 27 Sep 2023 13:47:41 +0200
    Ready: True
    Restart Count: 0
    Requests:
    cpu: 100m
    memory: 90Mi
    Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
    POD_NAME: nginx-ingress-nginx-controller-7995ddbbc4-fgh5g (v1:metadata.name)
    POD_NAMESPACE: ingress (v1:metadata.namespace)
    LD_PRELOAD: /usr/local/lib/libmimalloc.so
    Mounts:
    /usr/local/certificates/ from webhook-cert (ro)
    /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tvswc (ro)
    Conditions:
    Type Status
    Initialized True
    Ready True
    ContainersReady True
    PodScheduled True
    Volumes:
    webhook-cert:
    Type: Secret (a volume populated by a Secret)
    SecretName: nginx-ingress-nginx-admission
    Optional: false
    kube-api-access-tvswc:
    Type: Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds: 3607
    ConfigMapName: kube-root-ca.crt
    ConfigMapOptional:
    DownwardAPI: true
    QoS Class: Burstable
    Node-Selectors: kubernetes.io/os=linux
    Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists
    node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
    node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
    Events:
    Type Reason Age From Message

  ---

  Normal Scheduled 3m16s default-scheduler Successfully assigned ingress/nginx-ingress-nginx-controller-7995ddbbc4-fgh5g to aks-intmain-10577288-vmss000001
  Normal Pulling 3m17s kubelet Pulling image "registry.k8s.io/ingress-nginx/controller:v1.9.0@sha256:c15d1a617858d90fb8f8a2dd60b0676f2bb85c54e3ed11511794b86ec30c8c60"
  Normal Pulled 3m15s kubelet Successfully pulled image "registry.k8s.io/ingress-nginx/controller:v1.9.0@sha256:c15d1a617858d90fb8f8a2dd60b0676f2bb85c54e3ed11511794b86ec30c8c60" in 1.724907226s
  Normal Created 3m15s kubelet Created container controller
  Normal Started 3m15s kubelet Started container controller
  Normal RELOAD 3m12s nginx-ingress-controller NGINX reload triggered due to a change in configuration

  • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>
    Name: nginx-ingress-nginx-controller
    Namespace: ingress
    Labels: app.kubernetes.io/component=controller
    app.kubernetes.io/instance=nginx
    app.kubernetes.io/managed-by=Helm
    app.kubernetes.io/name=ingress-nginx
    app.kubernetes.io/part-of=ingress-nginx
    app.kubernetes.io/version=1.9.0
    helm.sh/chart=ingress-nginx-4.8.0
    Annotations: meta.helm.sh/release-name: nginx
    meta.helm.sh/release-namespace: ingress
    service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz
    Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx
    Type: LoadBalancer
    IP Family Policy: SingleStack
    IP Families: IPv4
    IP: 10.60.17.227
    IPs: 10.60.17.227
    LoadBalancer Ingress: 20.238.248.36
    Port: http 80/TCP
    TargetPort: http/TCP
    NodePort: http 32493/TCP
    Endpoints: 10.60.2.109:80
    Port: https 443/TCP
    TargetPort: https/TCP
    NodePort: https 30555/TCP
    Endpoints: 10.60.2.109:443
    Session Affinity: None
    External Traffic Policy: Local
    HealthCheck NodePort: 31434
    Events:

• Current state of ingress object, if applicable:

  • kubectl -n <appnnamespace> get all,ing -o wide
  • kubectl -n <appnamespace> describe ing <ingressname>
  • If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

• Others:

  • Any other related information like ;
    • copy/paste of the snippet (if applicable)
    • kubectl describe ... of any custom configmap(s) created and in use
    • Any other related information that may help

That is the ingress.yaml of the pod that now gets wrong requests:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: pgadmin-oauth
  namespace: tools
  annotations:
     cert-manager.io/acme-challenge-type: http01
     cert-manager.io/cluster-issuer: letsencrypt
     nginx.ingress.kubernetes.io/ssl-redirect: "true"
     nginx.ingress.kubernetes.io/use-regex: "true"
     nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-Script-Name /pgadmin;

How to reproduce this issue:

Anything else we need to know:

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[bmv126]

It may be because of #10393

You need to set below to true.
allow-snippet-annotations

[StefanLobbenmeierObjego]

That looks very likely, thanks! I will try tomorrow

[longwuyuan]

/remove-kind bug

[StefanLobbenmeierObjego]

Thanks, that was it ❤️

[kmasuhr]

It would be great if this breaking change was added to the changelog.

[xuxiaowei-com-cn]

[root@center-dev-1 ~]# kubectl -n ingress-nginx get deployments.apps ingress-nginx-controller -o yaml | grep configmap
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"controller","app.kubernetes.io/instance":"ingress-nginx","app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/part-of":"ingress-nginx","app.kubernetes.io/version":"1.9.1"},"name":"ingress-nginx-controller","namespace":"ingress-nginx"},"spec":{"minReadySeconds":0,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app.kubernetes.io/component":"controller","app.kubernetes.io/instance":"ingress-nginx","app.kubernetes.io/name":"ingress-nginx"}},"strategy":{"rollingUpdate":{"maxUnavailable":1},"type":"RollingUpdate"},"template":{"metadata":{"labels":{"app.kubernetes.io/component":"controller","app.kubernetes.io/instance":"ingress-nginx","app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/part-of":"ingress-nginx","app.kubernetes.io/version":"1.9.1"}},"spec":{"containers":[{"args":["/nginx-ingress-controller","--publish-service=$(POD_NAMESPACE)/ingress-nginx-controller","--election-id=ingress-nginx-leader","--controller-class=k8s.io/ingress-nginx","--ingress-class=nginx","--configmap=$(POD_NAMESPACE)/ingress-nginx-controller","--validating-webhook=:8443","--validating-webhook-certificate=/usr/local/certificates/cert","--validating-webhook-key=/usr/local/certificates/key"],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"LD_PRELOAD","value":"/usr/local/lib/libmimalloc.so"}],"image":"registry.k8s.io/ingress-nginx/controller:v1.9.1@sha256:605a737877de78969493a4b1213b21de4ee425d2926906857b98050f57a95b25","imagePullPolicy":"IfNotPresent","lifecycle":{"preStop":{"exec":{"command":["/wait-shutdown"]}}},"livenessProbe":{"failureThreshold":5,"httpGet":{"path":"/healthz","port":10254,"scheme":"HTTP"},"initialDelaySeconds":10,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1},"name":"controller","ports":[{"containerPort":80,"name":"http","protocol":"TCP"},{"containerPort":443,"name":"https","protocol":"TCP"},{"containerPort":8443,"name":"webhook","protocol":"TCP"}],"readinessProbe":{"failureThreshold":3,"httpGet":{"path":"/healthz","port":10254,"scheme":"HTTP"},"initialDelaySeconds":10,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1},"resources":{"requests":{"cpu":"100m","memory":"90Mi"}},"securityContext":{"allowPrivilegeEscalation":true,"capabilities":{"add":["NET_BIND_SERVICE"],"drop":["ALL"]},"runAsUser":101},"volumeMounts":[{"mountPath":"/usr/local/certificates/","name":"webhook-cert","readOnly":true}]}],"dnsPolicy":"ClusterFirst","nodeSelector":{"kubernetes.io/os":"linux"},"serviceAccountName":"ingress-nginx","terminationGracePeriodSeconds":300,"volumes":[{"name":"webhook-cert","secret":{"secretName":"ingress-nginx-admission"}}]}}}}
        - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
[root@center-dev-1 ~]# 

[root@center-dev-1 ~]# kubectl -n ingress-nginx get configmaps ingress-nginx-controller -o yaml
apiVersion: v1
data:
  allow-snippet-annotations: "true"
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"allow-snippet-annotations":"false"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"controller","app.kubernetes.io/instance":"ingress-nginx","app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/part-of":"ingress-nginx","app.kubernetes.io/version":"1.9.1"},"name":"ingress-nginx-controller","namespace":"ingress-nginx"}}
  creationTimestamp: "2023-10-13T02:25:27Z"
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.9.1
  name: ingress-nginx-controller
  namespace: ingress-nginx
  resourceVersion: "32089604"
  uid: 73614794-5de6-4bcd-968e-6ad874c68e20
[root@center-dev-1 ~]# 

[smanghi1]

if we set 'allow-snippet-annotations' value to true, won't we get CVE-2021-25742 as per this issue?
kubernetes/kubernetes#126811
or do we have any guidelines to use this directive?

[StefanLobbenmeierObjego]

@Lingkar since you eyed this:

The reason we implemented this kubernetes/kubernetes#126811