BUG: --enable-dynamic-configuration=true doesnt work with external auth

[mikksoone]

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

NGINX Ingress controller version:master (0.346)

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.2", GitCommit:"bdaeafa71f6c7c04636251031f93464384d54963", GitTreeState:"clean", BuildDate:"2017-10-24T19:48:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.10", GitCommit:"bebdeb749f1fa3da9e1312c4b08e439c404b3136", GitTreeState:"clean", BuildDate:"2017-11-03T16:31:49Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): Ubuntu
• Kernel (e.g. uname -a): 4.4.65-k8s Basic structure  #1 SMP Tue May 2 15:48:24 UTC 2017 x86_64 Linux
• Install tools: Kops

What happened:
External auth feature stopped working when nginx is started with --enable-dynamic-configuration=true

What you expected to happen:
External auth to work properly

How to reproduce it (as minimally and precisely as possible):

1. Create ingress file that uses auth-url
2. Make a request to a url that should use the auth-url in ingress

Anything else we need to know:

Dynamic configuration works for everything beside external auth. The backends are correctly configured and requests succeed.

How external auth is configured in ingress (the annotation has correct prefix, and localhost means that the external auth server is running as a sidecar for nginx)

    ingress.kubernetes.io/auth-url: http://localhost:8888/validate
    ingress.kubernetes.io/auth-response-headers: X-SEC-ID

Nginx failure log:

"POST /path HTTP/1.1" 200 0 "-" "Main (Main)" 0 0.975 [external-authentication] 127.0.0.1:8886 0 0.975 200 a158d8a6e08c556a1fad5c39aeed28f6
2018/03/21 21:06:12 [error] 178#178: *728 failed to run balancer_by_lua*: /etc/nginx/lua/balancer.lua:31: attempt to index local 'backend' (a nil value)
stack traceback:
	/etc/nginx/lua/balancer.lua:31: in function 'balance'
	/etc/nginx/lua/balancer.lua:97: in function 'call'
	balancer_by_lua:2: in function <balancer_by_lua:1> while connecting to upstream, client: IP, server: server, request: "POST /path HTTP/1.1", host: "host"

Nginx startup logs:

curl localhost:18080/configuration/backends
nil

More logs:

NGINX Ingress controller
  Release:    0.12.0
  Build:      git-0398c410
  Repository: https://github.com/aledbf/ingress-nginx
-------------------------------------------------------------------------------

W0321 19:50:09.284853       7 client_config.go:529] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0321 19:50:09.285014       7 main.go:181] Creating API client for https://REDUCTED:443
I0321 19:50:09.285641       7 main.go:201] trying to discover Kubernetes version
I0321 19:50:09.294493       7 main.go:225] Running in Kubernetes Cluster version v1.7 (v1.7.10) - git (clean) commit bebdeb749f1fa3da9e1312c4b08e439c404b3136 - platform linux/amd64
I0321 19:50:09.297059       7 main.go:84] validated nginx/nginx-default-backend as the default backend
I0321 19:50:09.299614       7 main.go:105] service nginx/ingress-nginx validated as source of Ingress status
I0321 19:50:09.549228       7 stat_collector.go:77] starting new nginx stats collector for Ingress controller running in namespace  (class nginx)
I0321 19:50:09.549244       7 stat_collector.go:78] collector extracting information from port 18080
I0321 19:50:09.560763       7 nginx.go:281] starting Ingress controller
I0321 19:50:09.572054       7 store.go:404] adding configmap nginx/ingress-nginx to backend
I0321 19:50:10.677349       7 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"app-bff-staging", Name:"app-bff-ingress", UID:"5f491870-20bc-11e8-889c-0a91ad1abda6", APIVersion:"extensions", ResourceVersion:"39561205", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress app-bff-staging/app-bff-ingress
...
I0321 19:50:10.762101       7 store.go:614] running initial sync of secrets
I0321 19:50:10.765779       7 backend_ssl.go:68] adding secret app-bff-staging/app-bff-staging-tls to the local store
I0321 19:50:10.767969       7 nginx.go:302] starting NGINX process...
I0321 19:50:10.768770       7 controller.go:183] backend reload required
I0321 19:50:10.768804       7 stat_collector.go:34] changing prometheus collector from  to vts
I0321 19:50:10.769177       7 util.go:64] system fs.file-max=2097152
I0321 19:50:10.769227       7 nginx.go:560] maximum number of open file descriptors : 523264
I0321 19:50:10.775212       7 leaderelection.go:174] attempting to acquire leader lease...
I0321 19:50:10.792038       7 status.go:196] new leader elected: ingress-nginx-2742315815-3kwgn
I0321 19:50:10.903607       7 nginx.go:658] NGINX configuration diff
I0321 19:50:10.903784       7 nginx.go:659] --- /etc/nginx/nginx.conf	2018-03-21 03:31:50.000000000 +0000
+++ /tmp/new-nginx-cfg724125397	2018-03-21 19:50:10.893639217 +0000
@@ -1,6 +1,1744 @@
-# A very simple nginx configuration file that forces nginx to start.
+
+daemon off;
+
+worker_processes 4;
+
 pid /run/nginx.pid;
 
-events {}
-http {}
-daemon off;
\ No newline at end of file
+worker_rlimit_nofile 523264;
+
+worker_shutdown_timeout 10s ;
+
+events {
+    multi_accept        on;
+    worker_connections  16384;
+    use                 epoll;
+}
+
+http {
+    lua_package_cpath "/usr/local/lib/lua/?.so;/usr/lib/x86_64-linux-gnu/lua/5.1/?.so;;";
+    lua_package_path "/etc/nginx/lua/?.lua;/etc/nginx/lua/vendor/?.lua;/usr/local/lib/lua/?.lua;;";
+
+    lua_shared_dict configuration_data 5M;
+    lua_shared_dict round_robin_state 1M;
+    lua_shared_dict locks 512k;
+
+    init_by_lua_block {
+        require("resty.core")
+        collectgarbage("collect")
+
+        -- init modules
+        local ok, res
+
+        ok, res = pcall(require, "configuration")
+        if not ok then
+          error("require failed: " .. tostring(res))
+        else
+          configuration = res
+        end
+
+        ok, res = pcall(require, "balancer")
+        if not ok then
+          error("require failed: " .. tostring(res))
+        else
+          balancer = res
+        end
+    }
+
+    init_worker_by_lua_block {
+        balancer.init_worker()
+    }
+
+    real_ip_header      proxy_protocol;
+
+    real_ip_recursive   on;
+
+    set_real_ip_from    0.0.0.0/0;
+
+    geoip_country       /etc/nginx/geoip/GeoIP.dat;
+    geoip_city          /etc/nginx/geoip/GeoLiteCity.dat;
+    geoip_org           /etc/nginx/geoip/GeoIPASNum.dat;
+    geoip_proxy_recursive on;
+
+    vhost_traffic_status_zone shared:vhost_traffic_status:10m;
+    vhost_traffic_status_filter_by_set_key $server_name;
+
+    aio                 threads;
+    aio_write           on;
+
+    tcp_nopush          on;
+    tcp_nodelay         on;
+
+    log_subrequest      on;
+
+    reset_timedout_connection on;
+
+    keepalive_timeout  75s;
+    keepalive_requests 100;
+
+    client_header_buffer_size       1k;
+    client_header_timeout           60s;
+    large_client_header_buffers     4 8k;
+    client_body_buffer_size         8k;
+    client_body_timeout             60s;
+
+    http2_max_field_size            4k;
+    http2_max_header_size           16k;
+
+    types_hash_max_size             2048;
+    server_names_hash_max_size      1024;
+    server_names_hash_bucket_size   64;
+    map_hash_bucket_size            64;
+
+    proxy_headers_hash_max_size     512;
+    proxy_headers_hash_bucket_size  64;
+
+    variables_hash_bucket_size      128;
+    variables_hash_max_size         2048;
+
+    underscores_in_headers          off;
+    ignore_invalid_headers          on;
+
+    limit_req_status                503;
+
+    include /etc/nginx/mime.types;
+    default_type text/html;
+
+    gzip on;
+    gzip_comp_level 5;
+    gzip_http_version 1.1;
+    gzip_min_length 256;
+    gzip_types application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component;
+    gzip_proxied any;
+    gzip_vary on;
+
+    # Custom headers for response
+
+    server_tokens on;
+
+    # disable warnings
+    uninitialized_variable_warn off;
+
+    # Additional available variables:
+    # $namespace
+    # $ingress_name
+    # $service_name
+    log_format upstreaminfo '%v - [$the_real_ip] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $request_id';
+
+    map $request_uri $loggable {
+
+        default 1;
+    }
+
+    access_log /var/log/nginx/access.log upstreaminfo if=$loggable;
+
+    error_log  /var/log/nginx/error.log info;
+
+    resolver REDUCTED valid=30s;
+
+    # Retain the default nginx handling of requests without a "Connection" header
+    map $http_upgrade $connection_upgrade {
+        default          upgrade;
+        ''               close;
+    }
+
+    map $http_x_forwarded_for $the_real_ip {
+
+        # Get IP address from Proxy Protocol
+        default          $proxy_protocol_addr;
+
+    }
+
+    # trust http_x_forwarded_proto headers correctly indicate ssl offloading
+    map $http_x_forwarded_proto $pass_access_scheme {
+        default          $http_x_forwarded_proto;
+        ''               $scheme;
+    }
+
+    # validate $pass_access_scheme and $scheme are http to force a redirect
+    map "$scheme:$pass_access_scheme" $redirect_to_https {
+        default          0;
+        "http:http"      1;
+        "https:http"     1;
+    }
+
+    map $http_x_forwarded_port $pass_server_port {
+        default           $http_x_forwarded_port;
+        ''                $server_port;
+    }
+
+    map $pass_server_port $pass_port {
+        443              443;
+        default          $pass_server_port;
+    }
+
+    # Obtain best http host
+    map $http_host $this_host {
+        default          $http_host;
+        ''               $host;
+    }
+
+    map $http_x_forwarded_host $best_http_host {
+        default          $http_x_forwarded_host;
+        ''               $this_host;
+    }
+
+    server_name_in_redirect off;
+    port_in_redirect        off;
+
+    rewrite_log             on;
+
+    ssl_protocols TLSv1.2;
+
+    # turn on session caching to drastically improve performance
+
+    ssl_session_cache builtin:1000 shared:SSL:10m;
+    ssl_session_timeout 10m;
+
+    # allow configuring ssl session tickets
+    ssl_session_tickets on;
+
+    # slightly reduce the time-to-first-byte
+    ssl_buffer_size 4k;
+
+    # allow configuring custom ssl ciphers
+    ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
+    ssl_prefer_server_ciphers on;
+
+    ssl_ecdh_curve auto;
+
+    proxy_ssl_session_reuse on;
+
+    upstream upstream_balancer {
+        server 0.0.0.1; # placeholder
+
+        balancer_by_lua_block {
+          balancer.call()
+        }
+
+        keepalive 1000;
+    }
+
+    ## start server _
+    server {
+        server_name _ ;
+
+        listen 80 proxy_protocol default_server  backlog=511;
+
+        listen [::]:80 proxy_protocol default_server  backlog=511;
+
+        set $proxy_upstream_name "-";
+
+        listen 443 proxy_protocol  default_server  backlog=511 ssl http2;
+
+        listen [::]:443 proxy_protocol  default_server  backlog=511 ssl http2;
+
+        # PEM sha: 06070084cadda62b8096e43ff43f464dfc4c57a7
+        ssl_certificate                         /ingress-controller/ssl/default-fake-certificate.pem;
+        ssl_certificate_key                     /ingress-controller/ssl/default-fake-certificate.pem;
+
+        location / {
+
+            if ($scheme = https) {
+            more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains;";
+            }
+
+            access_log off;
+
+            port_in_redirect off;
+
+            set $proxy_upstream_name "upstream-default-backend";
+
+            set $namespace      "";
+            set $ingress_name   "";
+            set $service_name   "";
+
+            client_max_body_size                    "1m";
+
+            proxy_set_header Host                   $best_http_host;
+
+            # Pass the extracted client certificate to the backend
+
+            proxy_set_header ssl-client-cert        "";
+            proxy_set_header ssl-client-verify      "";
+            proxy_set_header ssl-client-dn          "";
+
+            # Allow websocket connections
+            proxy_set_header                        Upgrade           $http_upgrade;
+
+            proxy_set_header                        Connection        $connection_upgrade;
+
+            proxy_set_header X-Real-IP              $the_real_ip;
+
+            proxy_set_header X-Forwarded-For        $the_real_ip;
+
+            proxy_set_header X-Forwarded-Host       $best_http_host;
+            proxy_set_header X-Forwarded-Port       $pass_port;
+            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
+            proxy_set_header X-Original-URI         $request_uri;
+            proxy_set_header X-Scheme               $pass_access_scheme;
+
+            # Pass the original X-Forwarded-For
+            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
+
+            # mitigate HTTPoxy Vulnerability
+            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
+            proxy_set_header Proxy                  "";
+
+            # Custom headers to proxied server
+
+            proxy_connect_timeout                   5s;
+            proxy_send_timeout                      60s;
+            proxy_read_timeout                      60s;
+
+            proxy_buffering                         "off";
+            proxy_buffer_size                       "4k";
+            proxy_buffers                           4 "4k";
+            proxy_request_buffering                 "on";
+
+            proxy_http_version                      1.1;
+
+            proxy_cookie_domain                     off;
+            proxy_cookie_path                       off;
+
+            # In case of errors try the next upstream server before returning an error
+            proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;
+
+            proxy_pass http://upstream_balancer;
+
+            proxy_redirect                          off;
+
+        }
+
+        # health checks in cloud providers require the use of port 80
+        location /healthz {
+            access_log off;
+            return 200;
+        }
+
+        # this is required to avoid error if nginx is being monitored
+        # with an external software (like sysdig)
+        location /nginx_status {
+            allow 127.0.0.1;
+            allow ::1;
+            deny all;
+
+            access_log off;
+            stub_status on;
+        }
+
+    }
+    ## end server _
+
+    ## start server REDUCTED
+    server {
+        server_name REDUCTED ;
+
+        listen 80 proxy_protocol;
+
+        listen [::]:80 proxy_protocol;
+
+        set $proxy_upstream_name "-";
+
+        listen 443 proxy_protocol  ssl http2;
+
+        listen [::]:443 proxy_protocol  ssl http2;
+
+        # PEM sha: 61ad3926d647a0de2c0fdad636f6c38e012a9dd3
+        ssl_certificate                         /ingress-controller/ssl/app-bff-staging-app-bff-staging-tls.pem;
+        ssl_certificate_key                     /ingress-controller/ssl/app-bff-staging-app-bff-staging-tls.pem;
+
+        location /.well-known/acme-challenge {
+
+            if ($scheme = https) {
+            more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains;";
+            }
+
+            port_in_redirect off;
+
+            set $proxy_upstream_name "kube-lego-kube-lego-nginx-8080";
+
+            set $namespace      "kube-lego";
+            set $ingress_name   "kube-lego-nginx";
+            set $service_name   "kube-lego-nginx";
+
+            client_max_body_size                    "1m";
+
+            proxy_set_header Host                   $best_http_host;
+
+            # Pass the extracted client certificate to the backend
+
+            proxy_set_header ssl-client-cert        "";
+            proxy_set_header ssl-client-verify      "";
+            proxy_set_header ssl-client-dn          "";
+
+            # Allow websocket connections
+            proxy_set_header                        Upgrade           $http_upgrade;
+
+            proxy_set_header                        Connection        $connection_upgrade;
+
+            proxy_set_header X-Real-IP              $the_real_ip;
+
+            proxy_set_header X-Forwarded-For        $the_real_ip;
+
+            proxy_set_header X-Forwarded-Host       $best_http_host;
+            proxy_set_header X-Forwarded-Port       $pass_port;
+            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
+            proxy_set_header X-Original-URI         $request_uri;
+            proxy_set_header X-Scheme               $pass_access_scheme;
+
+            # Pass the original X-Forwarded-For
+            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
+
+            # mitigate HTTPoxy Vulnerability
+            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
+            proxy_set_header Proxy                  "";
+
+            # Custom headers to proxied server
+
+            proxy_connect_timeout                   5s;
+            proxy_send_timeout                      60s;
+            proxy_read_timeout                      60s;
+
+            proxy_buffering                         "off";
+            proxy_buffer_size                       "4k";
+            proxy_buffers                           4 "4k";
+            proxy_request_buffering                 "on";
+
+            proxy_http_version                      1.1;
+
+            proxy_cookie_domain                     off;
+            proxy_cookie_path                       off;
+
+            # In case of errors try the next upstream server before returning an error
+            proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;
+
+            proxy_pass http://upstream_balancer;
+
+            proxy_redirect                          off;
+
+        }
+
+        location / {
+
+            if ($scheme = https) {
+            more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains;";
+            }
+
+            port_in_redirect off;
+
+            set $proxy_upstream_name "app-bff-staging-app-bff-service-3000";
+
+            set $namespace      "app-bff-staging";
+            set $ingress_name   "app-bff-notls-ingress";
+            set $service_name   "app-bff-service";
+
+            # enforce ssl on server side
+            if ($redirect_to_https) {
+
+                return 308 https://$best_http_host$request_uri;
+
+            }
+
+            client_max_body_size                    "1m";
+
+            proxy_set_header Host                   $best_http_host;
+
+            # Pass the extracted client certificate to the backend
+
+            proxy_set_header ssl-client-cert        "";
+            proxy_set_header ssl-client-verify      "";
+            proxy_set_header ssl-client-dn          "";
+
+            # Allow websocket connections
+            proxy_set_header                        Upgrade           $http_upgrade;
+
+            proxy_set_header                        Connection        $connection_upgrade;
+
+            proxy_set_header X-Real-IP              $the_real_ip;
+
+            proxy_set_header X-Forwarded-For        $the_real_ip;
+
+            proxy_set_header X-Forwarded-Host       $best_http_host;
+            proxy_set_header X-Forwarded-Port       $pass_port;
+            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
+            proxy_set_header X-Original-URI         $request_uri;
+            proxy_set_header X-Scheme               $pass_access_scheme;
+
+            # Pass the original X-Forwarded-For
+            proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
+
+            # mitigate HTTPoxy Vulnerability
+            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
+            proxy_set_header Proxy                  "";
+
+            # Custom headers to proxied server
+
+            proxy_connect_timeout                   5s;
+            proxy_send_timeout                      60s;
+            proxy_read_timeout                      60s;
+
+            proxy_buffering                         "off";
+            proxy_buffer_size                       "4k";
+            proxy_buffers                           4 "4k";
+            proxy_request_buffering                 "on";
+
+            proxy_http_version                      1.1;
+
+            proxy_cookie_domain                     off;
+            proxy_cookie_path                       off;
+
+            # In case of errors try the next upstream server before returning an error
+            proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;
+
+            proxy_pass http://upstream_balancer;
+
+            proxy_redirect                          off;
+
+        }
+
+    }
+    ## end server REDUCTED
+
...
+    # default server, used for NGINX healthcheck and access to nginx stats
+    server {
+        # Use the port 18080 (random value just to avoid known ports) as default port for nginx.
+        # Changing this value requires a change in:
+        # https://github.com/kubernetes/ingress-nginx/blob/master/controllers/nginx/pkg/cmd/controller/nginx.go
+        listen 18080 default_server  backlog=511;
+        listen [::]:18080 default_server  backlog=511;
+        set $proxy_upstream_name "-";
+
+        location /healthz {
+            access_log off;
+            return 200;
+        }
+
+        location /nginx_status {
+            set $proxy_upstream_name "internal";
+
+            access_log off;
+            stub_status on;
+
+        }
+
+        location /configuration {
+            allow 127.0.0.1;
+
+            allow ::1;
+
+            deny all;
+            content_by_lua_block {
+              configuration.call()
+            }
+        }
+
+        location / {
+
+            set $proxy_upstream_name "upstream-default-backend";
+
+            proxy_pass          http://upstream_balancer;
+
+        }
+
+    }
+}
+
+stream {
+    log_format log_stream [$time_local] $protocol $status $bytes_sent $bytes_received $session_time;
+
+    access_log /var/log/nginx/access.log log_stream;
+
+    error_log  /var/log/nginx/error.log;
+
+    # TCP services
+
+    # UDP services
+
+}
+
I0321 20:06:44.696023       7 controller.go:192] ingress backend successfully reloaded...
I0321 20:06:45.584802       7 backend_ssl.go:174] updating local copy of ssl certificate app-bff-staging/app-bff-staging-tls with missing intermediate CA certs
I0321 20:06:45.697116       7 nginx.go:771] posting backends configuration: [{"name":"app-bff-staging-app-bff-service-3000","service":{"metadata":{"name":"app-bff-service","namespace":"app-bff-staging","selfLink":"/api/v1/namespaces/app-bff-staging/services/app-bff-service","uid":"5f2f79ae-20bc-11e8-a1f9-02d01e4a667a","resourceVersion":"39561187","creationTimestamp":"2018-03-05T21:30:05Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"name\":\"app-bff-service\",\"namespace\":\"app-bff-staging\"},\"spec\":{\"ports\":[{\"name\":\"backend\",\"port\":3000,\"protocol\":\"TCP\",\"targetPort\":3000}],\"selector\":{\"app\":\"app-bff\"},\"type\":\"ClusterIP\"}}\n"}},"spec":{"ports":[{"name":"backend","protocol":"TCP","port":3000,"targetPort":3000}],"selector":{"app":"app-bff"},"clusterIP":"REDUCTED","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"port":3000,"secure":false,"secureCACert":{"secret":"","caFilename":"","pemSha":""},"sslPassthrough":false,"endpoints":[{"address":"REDUCTED","port":"3000","maxFails":0,"failTimeout":0,"target":{"kind":"Pod","namespace":"app-bff-staging","name":"app-bff-3-2881575126-jwllz","uid":"7643c83d-2cea-11e8-889c-0a91ad1abda6","resourceVersion":"42266422"}},{"address":"REDUCTED","port":"3000","maxFails":0,"failTimeout":0,"target":{"kind":"Pod","namespace":"app-bff-staging","name":"app-bff-stateless-3511195472-7p332","uid":"c4bbdd20-2cf2-11e8-889c-0a91ad1abda6","resourceVersion":"42276219"}},{"address":"REDUCTED","port":"3000","maxFails":0,"failTimeout":0,"target":{"kind":"Pod","namespace":"app-bff-staging","name":"app-bff-2-2331681453-cfwgn","uid":"65e3132c-2cea-11e8-889c-0a91ad1abda6","resourceVersion":"42266325"}},{"address":"REDUCTED","port":"3000","maxFails":0,"failTimeout":0,"target":{"kind":"Pod","namespace":"app-bff-staging","name":"app-bff-1-1759178922-bv2m0","uid":"c5898379-2cf2-11e8-889c-0a91ad1abda6","resourceVersion":"42276416"}}],"sessionAffinityConfig":{"name":"","cookieSessionAffinity":{"name":"","hash":""}}},{"name":"upstream-default-backend","service":{"metadata":{"name":"nginx-default-backend","namespace":"nginx","selfLink":"/api/v1/namespaces/nginx/services/nginx-default-backend","uid":"462f4e80-81ab-11e7-9380-0a36945459aa","resourceVersion":"42292594","creationTimestamp":"2017-08-15T11:17:07Z","labels":{"k8s-addon":"ingress-nginx.addons.k8s.io"},"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"k8s-addon\":\"ingress-nginx.addons.k8s.io\"},\"name\":\"nginx-default-backend\",\"namespace\":\"nginx\"},\"spec\":{\"ports\":[{\"port\":80,\"targetPort\":\"http\"}],\"selector\":{\"app\":\"nginx-default-backend\"}}}\n"}},"spec":{"ports":[{"protocol":"TCP","port":80,"targetPort":"http"}],"selector":{"app":"nginx-default-backend"},"clusterIP":"REDUCTED","type":"ClusterIP","sessionAffinity":"None"},"status":{"loadBalancer":{}}},"port":0,"secure":false,"secureCACert":{"secret":"","caFilename":"","pemSha":""},"sslPassthrough":false,"endpoints":[{"address":"REDUCTED","port":"8080","maxFails":0,"failTimeout":0,"target":{"kind":"Pod","namespace":"nginx","name":"nginx-default-backend-3184178138-hj7kt","uid":"b593b707-c2e3-11e7-95a9-06760f4b832a","resourceVersion":"21728925"}}],"sessionAffinityConfig":{"name":"","cookieSessionAffinity":{"name":"","hash":""}}}, REDUCTED_A_LOT_OF_OTHER_SERVICES]
 - [] - - [21/Mar/2018:20:06:45 +0000] "POST /configuration/backends HTTP/1.1" 201 5 "-" "Go-http-client/1.1" 12906 0.000 [-] - - - -
I0321 20:06:45.700599       7 controller.go:202] dynamic reconfiguration succeeded
I0321 20:06:46.606372       7 controller.go:183] backend reload required
I0321 20:06:46.606441       7 util.go:64] system fs.file-max=2097152
I0321 20:06:46.606449       7 nginx.go:560] maximum number of open file descriptors : 523264
I0321 20:06:46.811802       7 nginx.go:658] NGINX configuration diff
I0321 20:06:46.811823       7 nginx.go:659] --- /etc/nginx/nginx.conf	2018-03-21 20:06:44.633528996 +0000
...
I0321 20:06:46.958516       7 controller.go:192] ingress backend successfully reloaded...

Related issue: #2231

[oilbeater]

the annotation name should start with nginx

nginx.ingress.kubernetes.io/auth-url: http://localhost:8888/validate
nginx.ingress.kubernetes.io/auth-response-headers: X-SEC-ID

[mikksoone]

I tried to explicitly say in the comment that the annotation name is correct (Because the configmap has annotations-prefix: ingress.kubernetes.io)

[ElvinEfendi]

@mikksoone I could re-generate the issue locally and know what's the problem - it is because in athe subrequest for authentication proxy_upstream_name variable gets overridden. I'll open a PR with fix today or tomorrow and ping you there.