Getting real client ip using ingress on baremetal kubernetes without loadbalancer

[kskalski]

I'm using ingress-nginx configured by microk8s enable ingress, which roughly looks like that:

      - args:
        - /nginx-ingress-controller
        - --configmap=$(POD_NAMESPACE)/nginx-load-balancer-microk8s-conf
        - --tcp-services-configmap=$(POD_NAMESPACE)/nginx-ingress-tcp-microk8s-conf
        - --udp-services-configmap=$(POD_NAMESPACE)/nginx-ingress-udp-microk8s-conf
        - --enable-ssl-passthrough
...
        ports:
        - containerPort: 80
          hostPort: 80
          protocol: TCP
        - containerPort: 443
          hostPort: 443
          protocol: TCP
...
      dnsPolicy: ClusterFirstWithHostNet
      hostNetwork: true

that is exposed to the internet and is serving some http/https traffic routing to my services (this set-up seems to be described here as "host network" https://kubernetes.github.io/ingress-nginx/deploy/baremetal/).
Surprisingly when looking at headers of the requests arriving at my server, the IPs provided are meaningless:

GET /echo HTTP/1.1
Host: dev.glosowanie.net
X-Request-ID: 432f84f2d96e4311ac92d8f0b2344b43
X-Real-IP: 127.0.0.1
X-Forwarded-For: 127.0.0.1
X-Forwarded-Host: dev.glosowanie.net
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Scheme: https
cache-control: max-age=0
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36

Originally the ingress controller config also had --publish-status-address=127.0.0.1 flag, which I already removed, but it didn't make much difference, the requests are still getting something like above 127.0.0.1 in headers or are reported using cni0 IP in logs.

I would like to get the proper X-Forwarder-Host / X-Real-IP from originator of the request, so I can log it in my server or even use inside ingress for other features like 'nginx.ingress.kubernetes.io/whitelist-source-range' annotation (right now it is seems to be provided the address of cni0 or --publish-status-address)

Is the use of 'host network' somehow interfering with headers? Should I experiment with flags from https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/? It seems that the IP is garbled even when I use nginx.ingress.kubernetes.io/ssl-passthrough: "true" and my GRPCS backend, which is super-weird, since I though in this mode ingress-nginx won't parse/modify headers at all. Is this unrelated to the ingress-nginx and cause by other parts of microk8s kubernetes layout?

/triage support

[kskalski]

It seems to be related to
#672
and fortunately setting
use-proxy-protocol: true
in nginx-load-balancer-microk8s-conf ConfigMap solved my problem even though I don't use any load balancer in front of ingress-nginx

[kskalski]

Additional issue is that this doesn't work with ssl-passthrough enabled (someone here had similar problem: rancher/rancher#27018). I suppose this mode disables any header processing, even adding new ones...

I wonder though, if this is the case, is there a way to configure ingress to do a kind of TCP-passthrough? If ingress-nginx actually intercepts the SSL connection passing it to the backend, but can't add or process headers, would it be possible to make it work as totally opaque proxy avoding the backend even noticing that packets went through it.

[benoist]

We have a similar setup and figured out the correct setting for the following setup:

FW -> MetalLB -> Ingress Controller (with --ssl-passthrough)

We needed to set the following flags:

enable-real-ip: true
forwarded-for-header: proxy_protocol

I was looking at the code and it turns out the Go proxy is using the proxy_protocol to proxy requests to nginx.

This is not to be mistaken with the use-proxy-protocol, because that will also instruct the Go proxy to use the proxy protocol which is not what we want.
We want the Go proxy to use the remote_addr (the default), the go proxy already uses the proxy protocol. Nginx by default expects the X-Forwarded-For which works with L7 load balancers, so using the forwarded-for-header with proxy_protocol only changes what is used for the X-Real-IP header and gives us the desired result

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[kskalski]

Thanks, setting enable-real-ip and forwarded-for-header: proxy_protocol seems to work for me when I don't set use-proxy-protocol. Hopefully it will fix the problem I was having with use-proxy-protocol, which prevented cert manager from validating domains.

[gadekarnitesh]

@kskalski is it possible to get real client IP for UDP>
I am create the ingress controller using following properties but not working.
controller: ingressClass: udp-ingress proxySetHeaders: proxy-protocol: "True" real-ip-header: "proxy_protocol" service: enableHttp: false enableHttps: false externalTrafficPolicy: Local udp: 162: "default/snmp:162"