Upgrade to 1.9 causes "unknown variable" errors in authentication configuration

[chazapis]

What happened:

I upgraded my cluster using Helm chart 4.8.0, which uses ingress-nginx 1.9.0.

I have several services authenticating to an OAuth server using Vouch Proxy, as described here.

This is part of an ingress entry:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/auth-response-headers: X-Vouch-User
    nginx.ingress.kubernetes.io/auth-signin: https://<OAuth server URL>/login?url=$scheme://$http_host$request_uri&vouch-failcount=$auth_resp_failcount&X-Vouch-Token=$auth_resp_jwt&error=$auth_resp_err
    nginx.ingress.kubernetes.io/auth-snippet: |-
      auth_request_set $auth_resp_jwt $upstream_http_x_vouch_jwt;
      auth_request_set $auth_resp_err $upstream_http_x_vouch_err;
      auth_request_set $auth_resp_failcount $upstream_http_x_vouch_failcount;
    nginx.ingress.kubernetes.io/auth-url: https://<OAuth server URL>/validate
    nginx.ingress.kubernetes.io/configuration-snippet: |2-
        auth_request_set $auth_resp_x_vouch_username $upstream_http_x_vouch_idp_claims_preferred_username;
        access_by_lua_block {
          if not (string.match(ngx.var.auth_resp_x_vouch_username, "<username>")) then
            ngx.exit(ngx.HTTP_FORBIDDEN);
          end
        }
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
  name: hello-kubernetes
  namespace: <namespace>
spec:
  ...

Which causes the following errors to be repeatedly reported in the log:

E0927 14:16:21.557171       7 controller.go:205] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: exit status 1
2023/09/27 14:16:21 [emerg] 24#24: unknown "auth_resp_failcount" variable
nginx: [emerg] unknown "auth_resp_failcount" variable
nginx: configuration file /tmp/nginx/nginx-cfg4035240519 test failed

-------------------------------------------------------------------------------
E0927 14:16:21.557227       7 queue.go:131] "requeuing" err=<
	
	-------------------------------------------------------------------------------
	Error: exit status 1
	2023/09/27 14:16:21 [emerg] 24#24: unknown "auth_resp_failcount" variable
	nginx: [emerg] unknown "auth_resp_failcount" variable
	nginx: configuration file /tmp/nginx/nginx-cfg4035240519 test failed
	
	-------------------------------------------------------------------------------
 > key="initial-sync"
I0927 14:16:21.557278       7 event.go:298] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-ingress-nginx-controller-858cf5c756-8xq6c", UID:"6977ce41-4682-4fc0-a64e-51b17022033d", APIVersion:"v1", ResourceVersion:"13542", FieldPath:""}): type: 'Warning' reason: 'RELOAD' Error reloading NGINX: 
-------------------------------------------------------------------------------
Error: exit status 1
2023/09/27 14:16:21 [emerg] 24#24: unknown "auth_resp_failcount" variable
nginx: [emerg] unknown "auth_resp_failcount" variable
nginx: configuration file /tmp/nginx/nginx-cfg4035240519 test failed

-------------------------------------------------------------------------------
I0927 14:16:24.788026       7 controller.go:190] "Configuration changes detected, backend reload required"

What you expected to happen:

Reverting to Helm chart 4.7.2, which uses ingress-nginx 1.8.2 solves the issue. I see no errors and the ingress entry works correctly.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.9.0
  Build:         4bd3d6b8a00b01b009f225a5593ce502cce5c26b
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

-------------------------------------------------------------------------------

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.2", GitCommit:"7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647", GitTreeState:"clean", BuildDate:"2023-05-17T14:20:07Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.2", GitCommit:"7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647", GitTreeState:"clean", BuildDate:"2023-05-17T14:13:28Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"linux/arm64"}

Environment:

• Cloud provider or hardware configuration: Using an 2020 M1 MacBook Pro

• OS (e.g. from /etc/os-release): macOS 13.5

• Kernel (e.g. uname -a):

• Install tools: Kubernetes 1.27.2 provided by Docker Desktop 4.23.0

  • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.

• Basic cluster related info:

  • kubectl version
  • kubectl get nodes -o wide

• How was the ingress-nginx-controller installed:

  • If helm was used then please show output of helm ls -A | grep -i ingress

ingress                                         	ingress-nginx	5       	2023-09-27 17:16:17.083052 +0300 EEST  	deployed	ingress-nginx-4.8.0         	1.9.0

• If helm was used then please show output of helm -n <ingresscontrollernamepspace> get values <helmreleasename>

USER-SUPPLIED VALUES:
controller:
  admissionWebhooks:
    enabled: false
  extraArgs:
    default-ssl-certificate: ingress-nginx/ssl-certificate
  ingressClassResource:
    default: true
  watchIngressWithoutClass: true

• If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used

• if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances

• Current State of the controller:

  • kubectl describe ingressclasses
  • kubectl -n <ingresscontrollernamespace> get all -A -o wide
  • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
  • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>

• Current state of ingress object, if applicable:

  • kubectl -n <appnnamespace> get all,ing -o wide
  • kubectl -n <appnamespace> describe ing <ingressname>
  • If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

• Others:

  • Any other related information like ;
    • copy/paste of the snippet (if applicable)
    • kubectl describe ... of any custom configmap(s) created and in use
    • Any other related information that may help

How to reproduce this issue:

Anything else we need to know:

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[bmv126]

It may be because of #10393

You need to set below to true.
allow-snippet-annotations

[chazapis]

Thanks!

I confirm that setting the following variable fixed the issue:

controller:
  allowSnippetAnnotations: true