ingress-nginx is adding HTTP_ prefix to all the request headers

[rohan-97]

What happened:

Hello,

I tried to setup a kubernetes cluter with ingress-nginx ingress controller.
the ingress controller comes packaged with Microk8s installer and is enabled as a Microk8s plugin.

the issue that I am facing is that all the request header which are send from outside world to the upstream backend endpoints have HTTP_ prefix attached.

I have created an echoserver app which acts as an endpoint and returns the request headers that it receives, and following is the response

local_admin@rohannode:~$ curl --user admin:admin123 -H "theKey:theValue" http://localhost/endpoint

Hello, From endpoint! headers received : {'ACTUAL_SERVER_PROTOCOL': 'HTTP/1.1', 'PATH_INFO': '/endpoint', 'QUERY_STRING': '', 'REMOTE_ADDR': '10.1.206.80', 'REMOTE_PORT': '54972', 'REQUEST_METHOD': 'GET', 'REQUEST_URI': '/endpoint', 'SCRIPT_NAME': '', 'SERVER_NAME': 'localhost', 'SERVER_PROTOCOL': 'HTTP/1.1', 'SERVER_SOFTWARE': 'Cheroot/unknown Server', 'wsgi.errors': <_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>, 'wsgi.input': <cheroot.server.KnownLengthRFile object at 0x7f431c1ce850>, 'wsgi.input_terminated': False, 'wsgi.multiprocess': False, 'wsgi.multithread': True, 'wsgi.run_once': False, 'wsgi.url_scheme': 'http', 'wsgi.version': (1, 0), 'SERVER_PORT': '4567', 'HTTP_HOST': 'localhost', 'HTTP_X_REQUEST_ID': 'b501c21944ab5388f476c920ec16904d', 'HTTP_X_REAL_IP': '10.69.37.249', 'HTTP_X_FORWARDED_FOR': '10.69.37.249', 'HTTP_X_FORWARDED_HOST': 'localhost', 'HTTP_X_FORWARDED_PORT': '80', 'HTTP_X_FORWARDED_PROTO': 'http', 'HTTP_X_FORWARDED_SCHEME': 'http', 'HTTP_X_SCHEME': 'http', 'HTTP_AUTHORIZATION': 'Basic YWRtaW46YWRtaW4xMjM=', 'HTTP_USER_AGENT': 'curl/7.74.0', 'HTTP_ACCEPT': '*/*', 'HTTP_THEKEY': 'theValue'}

Please note that the headers passed in curl request is theKey, however in resposne we get HTTP_THEKEY, which should be theKey

What you expected to happen:

Expected behaviour is that request headers should not be tampered and no prefix should be added to request header

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

bash-5.1$ /nginx-ingress-controller --version

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.5.1
  Build:         d003aae913cc25f375deb74f898c7f3c65c06f05
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

-------------------------------------------------------------------------------

Kubernetes version (use kubectl version):

local_admin@rohannode:~/af-servicedispatcher/auth_server$ sudo kubectl version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.2

Environment:

• Cloud provider or hardware configuration: 16GB Ram 100GB SSD 8CPU

• OS (e.g. from /etc/os-release): Debian GNU/Linux 11 (bullseye)

• Kernel (e.g. uname -a): Linux rohannode 6.1.22generic Basic structure  #1 SMP PREEMPT_DYNAMIC Wed Apr 12 19:17:46 UTC 2023 x86_64 GNU/Linux

• Install tools: Microk8s

  • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.

• Basic cluster related info:

  • kubectl version

  Client Version: v1.28.2
  Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
  Server Version: v1.28.2
  
  
  

• How was the ingress-nginx-controller installed:

  • If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used
    sudo microk8s enable ingress

• Current State of the controller:

  • kubectl describe ingressclasses

local_admin@rohannode:~/af-servicedispatcher/auth_server$ sudo kubectl describe ingressclasses
Name:         public
Labels:       <none>
Annotations:  ingressclass.kubernetes.io/is-default-class: true
Controller:   k8s.io/ingress-nginx
Events:       <none>


Name:         nginx
Labels:       <none>
Annotations:  <none>
Controller:   k8s.io/ingress-nginx
Events:       <none>

• kubectl -n <ingresscontrollernamespace> get all -o wide

local_admin@rohannode:~/af-servicedispatcher/auth_server$ sudo kubectl -n ingress get all  -o wide
NAME                                          READY   STATUS    RESTARTS        AGE   IP            NODE        NOMINATED NODE   READINESS GATES
pod/nginx-ingress-microk8s-controller-vxh92   1/1     Running   10 (2d3h ago)   16d   10.1.206.80   rohannode   <none>           <none>

NAME                                               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE   CONTAINERS               IMAGES                                            SELECTOR
daemonset.apps/nginx-ingress-microk8s-controller   1         1         1       1            1           <none>          16d   nginx-ingress-microk8s   registry.k8s.io/ingress-nginx/controller:v1.5.1   name=nginx-ingress-microk8s

• kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>

local_admin@rohannode:~/af-servicedispatcher/auth_server$ sudo kubectl -n ingress describe po nginx-ingress-microk8s-controller-vxh92
Name:             nginx-ingress-microk8s-controller-vxh92
Namespace:        ingress
Priority:         0
Service Account:  nginx-ingress-microk8s-serviceaccount
Node:             rohannode/10.69.37.249
Start Time:       Wed, 11 Oct 2023 09:47:52 +0000
Labels:           controller-revision-hash=58857899d8
                  name=nginx-ingress-microk8s
                  pod-template-generation=1
Annotations:      cni.projectcalico.org/containerID: 7e0a3900d86c0039e9c2f126033735265d714a062e7928f83c9bf37e3514153c
                  cni.projectcalico.org/podIP: 10.1.206.80/32
                  cni.projectcalico.org/podIPs: 10.1.206.80/32
Status:           Running
IP:               10.1.206.80
IPs:
  IP:           10.1.206.80
Controlled By:  DaemonSet/nginx-ingress-microk8s-controller
Containers:
  nginx-ingress-microk8s:
    Container ID:  containerd://2a0492e2b79a2be1405ba9b806c582f535551b7ba3a110441494733004cb5bd8
    Image:         registry.k8s.io/ingress-nginx/controller:v1.5.1
    Image ID:      docker.io/library/import-2023-10-11@sha256:4ba73c697770664c1e00e9f968de14e08f606ff961c76e5d7033a4a9c593c629
    Ports:         80/TCP, 443/TCP, 10254/TCP
    Host Ports:    80/TCP, 443/TCP, 10254/TCP
    Args:
      /nginx-ingress-controller
      --configmap=$(POD_NAMESPACE)/nginx-load-balancer-microk8s-conf
      --tcp-services-configmap=$(POD_NAMESPACE)/nginx-ingress-tcp-microk8s-conf
      --udp-services-configmap=$(POD_NAMESPACE)/nginx-ingress-udp-microk8s-conf
      --ingress-class=public
      --default-ssl-certificate=default/tls-secret
      --publish-status-address=127.0.0.1
    State:          Running
      Started:      Wed, 25 Oct 2023 11:53:32 +0000
    Last State:     Terminated
      Reason:       StartError
      Message:      failed to start containerd task "e6e0943ebfa801ce00cdee15ca25d34e3035ad78e5bca5b21be261d2135070f3": context canceled: unknown
      Exit Code:    128
      Started:      Thu, 01 Jan 1970 00:00:00 +0000
      Finished:     Wed, 25 Oct 2023 11:51:22 +0000
    Ready:          True
    Restart Count:  10
    Liveness:       http-get http://:10254/healthz delay=10s timeout=5s period=10s #success=1 #failure=3
    Readiness:      http-get http://:10254/healthz delay=0s timeout=5s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       nginx-ingress-microk8s-controller-vxh92 (v1:metadata.name)
      POD_NAMESPACE:  ingress (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9lmpr (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  kube-api-access-9lmpr:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:                      <none>

How to reproduce this issue:

As minimally and precisely as possible. Keep in mind we do not have access to your cluster or application.
Help up us (if possible) reproducing the issue using minikube or kind.

Install Microk8s

• Microk8s: https://microk8s.io/docs/getting-started

Install the ingress controller

sudo microk8s enable ingress

Install an application that will act as default backend (is just an echo app)

kubectl apply -f echoserver.yaml

Content of echoserver.yaml is as follows

# Source: chart/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: echo-server
  name: echo-server
  namespace: my-echoserver
spec:
  ports:
    - port: 4567
      targetPort: 4567
  selector:
    app: echo-server
---
# Source: chart/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-server
  namespace: my-echoserver
  labels:
    app: echo-server
spec:
  selector:
    matchLabels:
      app: echo-server
  template:
    metadata:
      labels:
        app: echo-server
    spec:
      containers:
      - name: echo-server
        image: localhost:32000/target-app:latest
        imagePullPolicy:
        ports:
          - containerPort: 4567
      dnsConfig:
        options:
          - name: ndots
            value: "4"
---
# Source: chart/templates/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: echo-server
  namespace: my-echoserver
  annotations:
    nginx.ingress.kubernetes.io/auth-response-headers: X-Correlation-Id
    nginx.ingress.kubernetes.io/auth-url: "http://auth-server-service.my-service-dispatcher.svc.cluster.local/auth"

spec:
  rules:
    - http:
        paths:
          - path: /endpoint
            pathType: Prefix
            backend:
              service:
                name: echo-server
                port:
                  number: 4567

make a request

local_admin@rohannode:~$ curl --user admin:admin123 -H "theKey:theValue" http://localhost/endpoint

Hello, From endpoint! headers received : {'ACTUAL_SERVER_PROTOCOL': 'HTTP/1.1', 'PATH_INFO': '/endpoint', 'QUERY_STRING': '', 'REMOTE_ADDR': '10.1.206.80', 'REMOTE_PORT': '54972', 'REQUEST_METHOD': 'GET', 'REQUEST_URI': '/endpoint', 'SCRIPT_NAME': '', 'SERVER_NAME': 'localhost', 'SERVER_PROTOCOL': 'HTTP/1.1', 'SERVER_SOFTWARE': 'Cheroot/unknown Server', 'wsgi.errors': <_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>, 'wsgi.input': <cheroot.server.KnownLengthRFile object at 0x7f431c1ce850>, 'wsgi.input_terminated': False, 'wsgi.multiprocess': False, 'wsgi.multithread': True, 'wsgi.run_once': False, 'wsgi.url_scheme': 'http', 'wsgi.version': (1, 0), 'SERVER_PORT': '4567', 'HTTP_HOST': 'localhost', 'HTTP_X_REQUEST_ID': 'b501c21944ab5388f476c920ec16904d', 'HTTP_X_REAL_IP': '10.69.37.249', 'HTTP_X_FORWARDED_FOR': '10.69.37.249', 'HTTP_X_FORWARDED_HOST': 'localhost', 'HTTP_X_FORWARDED_PORT': '80', 'HTTP_X_FORWARDED_PROTO': 'http', 'HTTP_X_FORWARDED_SCHEME': 'http', 'HTTP_X_SCHEME': 'http', 'HTTP_AUTHORIZATION': 'Basic YWRtaW46YWRtaW4xMjM=', 'HTTP_USER_AGENT': 'curl/7.74.0', 'HTTP_ACCEPT': '*/*', 'HTTP_THEKEY': 'theValue'}

The echoserver returns the headers as part of resposne.

Please note that the headers passed in curl request is theKey, however in resposne we get HTTP_THEKEY, which should be theKey

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

We don't do CI test on microk8s.

I tried reproducing. Is my reproduce effort wrong ?

[longwuyuan]

/remove-kind bug

[rohan-97]

Hello @longwuyuan ,

Thanks for looking into the issue and for quick response.
I was also debugging the issue and I found that the issue is not with ingress controller but with the endpoint that I was Hitting.
The endpoint was implementd using Python's webpy framework.

After comparing with other technologies I found that webpy was adding HTTP_ prefix in the header and was returning the headers back to the client.
And I doubted that the headers are translated through ingress controller.

Apologies for the issue created but seems like the issue is not valid.

For the record I am attaching response of headers implemented by different technologies

1. Python Webpy

local_admin@rohannode:~/project_echoserver$ curl --user admin:admin123 -H "testheader:rohan1234" http://localhost/api/v1/echoserver/webpy
{
    "ACTUAL_SERVER_PROTOCOL": "HTTP/1.1",
    "PATH_INFO": "/api/v1/echoserver/webpy",
    "QUERY_STRING": "",
    "REMOTE_ADDR": "10.1.206.89",
    "REMOTE_PORT": "49214",
    "REQUEST_METHOD": "GET",
    "REQUEST_URI": "/api/v1/echoserver/webpy",
    "SCRIPT_NAME": "",
    "SERVER_NAME": "localhost",
    "SERVER_PROTOCOL": "HTTP/1.1",
    "SERVER_SOFTWARE": "Cheroot/9.0.0 Server",
    "wsgi.errors": "<_io.TextIOWrapper name=\"<stderr>\" mode=\"w\" encoding=\"utf-8\">",
    "wsgi.input": "<cheroot.server.KnownLengthRFile object at 0x7f1b2ce07a50>",
    "wsgi.input_terminated": false,
    "wsgi.multiprocess": false,
    "wsgi.multithread": true,
    "wsgi.run_once": false,
    "wsgi.url_scheme": "http",
    "wsgi.version": [1,0],
    "SERVER_PORT": "80",
    "HTTP_X_CORRELATION_ID": "theValue",
    "HTTP_HOST": "localhost",
    "HTTP_X_REQUEST_ID": "69eb8bf54532bd08d48cd7d6120f684c",
    "HTTP_X_REAL_IP": "10.69.37.249",
    "HTTP_X_FORWARDED_FOR": "10.69.37.249",
    "HTTP_X_FORWARDED_HOST": "localhost",
    "HTTP_X_FORWARDED_PORT": "80",
    "HTTP_X_FORWARDED_PROTO": "http",
    "HTTP_X_FORWARDED_SCHEME": "http",
    "HTTP_X_SCHEME": "http",
    "HTTP_TEST": "abcd",
    "HTTP_AUTHORIZATION": "Basic YWRtaW46YWRtaW4xMjM=",
    "HTTP_USER_AGENT": "curl/7.74.0",
    "HTTP_ACCEPT": "*/*"
}

2. Flask

local_admin@rohannode:~/project_echoserver$ curl --user admin:admin123 -H "theKey:theValue" http://localhost/api/v1/echoserver/flask
{
  "Accept": "*/*",
  "Authorization": "Basic YWRtaW46YWRtaW4xMjM=",
  "Host": "localhost",
  "Test": "abcd",
  "Thekey": "theValue",
  "User-Agent": "curl/7.74.0",
  "X-Correlation-Id": "973cd15d-51c1-4139-9a2e-daf27ce7753f",
  "X-Forwarded-For": "10.69.37.249",
  "X-Forwarded-Host": "localhost",
  "X-Forwarded-Port": "80",
  "X-Forwarded-Proto": "http",
  "X-Forwarded-Scheme": "http",
  "X-Real-Ip": "10.69.37.249",
  "X-Request-Id": "24951ed31adca6030b413cbee9d14816",
  "X-Scheme": "http"
}

3. Node JS

local_admin@rohannode:~/project_echoserver/node$ curl --user admin:admin123 -H "X-Correlation-Id:theValue" http://localhost/api/v1/echoserver/node | python3 -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   563  100   563    0     0  13404      0 --:--:-- --:--:-- --:--:-- 13093
{
    "x-correlation-id": "theValue",
    "host": "localhost",
    "x-request-id": "f44c053a3f3bccc900adc274137b1ccc",
    "x-real-ip": "10.69.37.249",
    "x-forwarded-for": "10.69.37.249",
    "x-forwarded-host": "localhost",
    "x-forwarded-port": "80",
    "x-forwarded-proto": "http",
    "x-forwarded-scheme": "http",
    "x-scheme": "http",
    "test": "abcd",
    "authorization": "Basic YWRtaW46YWRtaW4xMjM=",
    "user-agent": "curl/7.74.0",
    "accept": "*/*"
}

4. GO

local_admin@rohannode:~/af-servicedispatcher/auth_server$ curl --user admin:admin123 -H "x-correlation-id:rohan1234" http://localhost/api/v1/echoserver | python3 -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1622  100  1622    0     0  10532      0 --:--:-- --:--:-- --:--:-- 10464
{
    "host": {
        "hostname": "localhost",
        "ip": "10.1.206.89",
        "ips": []
    },
    "http": {
        "method": "GET",
        "baseUrl": "",
        "originalUrl": "/param?query=demo",
        "protocol": "http"
    },
    "request": {
        "params": {
            "0": "/param"
        },
        "query": {
            "query": "demo"
        },
        "cookies": {},
        "body": {},
        "headers": {
            "x-correlation-id": "rohan1234",
            "host": "localhost",
            "x-request-id": "7728554d465a317668eee20bf8e364e3",
            "x-real-ip": "10.69.37.249",
            "x-forwarded-for": "10.69.37.249",
            "x-forwarded-host": "localhost",
            "x-forwarded-port": "80",
            "x-forwarded-proto": "http",
            "x-forwarded-scheme": "http",
            "x-scheme": "http",
            "test": "abcd",
            "authorization": "Basic YWRtaW46YWRtaW4xMjM=",
            "user-agent": "curl/7.74.0",
            "accept": "*/*"
        }
    },
    "environment": {
        "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
        "HOSTNAME": "echo-server-67c7758777-4252d",
        "NODE_VERSION": "20.9.0",
        "YARN_VERSION": "1.22.19",
        "KUBERNETES_PORT": "tcp://10.152.183.1:443",
        "KUBERNETES_PORT_443_TCP": "tcp://10.152.183.1:443",
        "ECHO_SERVER_PORT_80_TCP_PORT": "80",
        "KUBERNETES_SERVICE_PORT": "443",
        "ECHO_SERVER_PORT_80_TCP": "tcp://10.152.183.125:80",
        "ECHO_SERVER_PORT_80_TCP_PROTO": "tcp",
        "KUBERNETES_SERVICE_HOST": "10.152.183.1",
        "KUBERNETES_PORT_443_TCP_PROTO": "tcp",
        "KUBERNETES_PORT_443_TCP_PORT": "443",
        "KUBERNETES_PORT_443_TCP_ADDR": "10.152.183.1",
        "ECHO_SERVER_SERVICE_HOST": "10.152.183.125",
        "ECHO_SERVER_PORT": "tcp://10.152.183.125:80",
        "KUBERNETES_SERVICE_PORT_HTTPS": "443",
        "ECHO_SERVER_SERVICE_PORT": "80",
        "ECHO_SERVER_PORT_80_TCP_ADDR": "10.152.183.125",
        "HOME": "/root"
    }
}

It turns out that different implementation interprets headers differently and because of behaviour of WebPY framework I doubted the ingress controller.

Thanks a lot for the quick resposne.
Closing this issue

[longwuyuan]

🙏