Use tproxy to allow preservation of IP addresses

[plwhite]

Feature Request

What problem are you trying to solve?

When traffic arrives at a container in a pod via Linked in, the IP addresses are mangled.

• The pod has to listen on localhost instead of the pod IP (which is a minor issue, but does mean that behaviour may need to differ based on whether a service mesh is present).

• All incoming traffic appears to come from localhost. This impacts logging; can break certain protocols which mandate behaviour based on IP address in use (SIP); and stops blacklisting of IPs for security.

I'd like to address the second of these.

How should the problem be solved?

The most likely way to fix this is to use TPROXY in iptables rules. Roughly this involves:

• changes to iptables rules;

• giving the proxy container CAP_NET_ADMIN privileges;

• changing the proxy container to maintain the source address.

This is an option for istio, with some interesting discussion here: istio/istio#4654.

Any alternatives you've considered?

I haven't been able to think of any, though open to suggestions.

How would users interact with this feature?

The use of TPROXY would have to be a Linkerd option, enabled probably globally, both for security reasons (CAP_NET_ADMIN) and because of environment specific issues with kernel modules such as istio/istio#22500

[olix0r]

@plwhite This is definitely a reasonable request; but it has some big implications, so I think we'd want someone to put together an RFC that outlines a plan of attack. Such an RFC will need, at least, some attention to the following:

• Before installation, how can we know whether a cluster supports this? Browsing the Istio backlog, it appears as if some operating systems (like GKE COS) may not support this? (Basically, how do we need to modify linkerd check --pre)
• Can this be instrumented via the CNI in addition to the proxy-init container?
• What logic does the proxy need to implement (in terms of system calls) to support this?
• Finally, we'd need to work through the configuration surface area--particularly the Helm configuration and how that configures the proxy injector, CNI, etc.

If I were going about implementing this, I'd probably hack the proxy-init and proxy to prove the core functionality quickly before getting too deep on the plan. And then we'll probably want to take care to ensure that this lands with lands as an experimental feature that doesn't put any burden on users who have not enabled it.

[hochuenw-dd]

Hi @plwhite @olix0r ,
I'm not very familiar with iptables, but looks like linkerd is using iptables REDIRECT, and REDIRECT per my understanding only changes the destination IP/Port and keeps source IP/Port.

All incoming traffic appears to come from localhost for linkerd proxy

Is this from the view of the application or linkerd proxy? My understanding is that linkerd proxy can see the real source ip, but maybe not the application. If this is correct, why would introducing tproxy help the main container see the source ip? Thanks

[plwhite]

It's true that the traffic arrives at the proxy with the right source IP, but the workload container sees localhost. Think about incoming traffic to the pod, say from 1.2.3.4 to 4.3.2.1.

• The proxy listens for incoming traffic on the 4.3.2.1 address, on a well known proxy port.
• The workload container listens on localhost.
• A REDIRECT happens so the traffic arrives at the proxy port. The proxy sees 1.2.3.4 as the source and the destination as 4.3.2.1 but with the port changed (and the proxy has access to the original port anyway).
• The proxy does its processing, and then sends the data to the workload container. However, that connection cannot have source 1.2.3.4 since that is not a valid IP in this pod, and so the proxy has to create a new connection which has both source IP and destination IP as localhost.

If you use TPROXY, then the proxy (which has all the information to hand) can create a proxy connection with the correct source IP. This requires not just changes to iptables config, but to the proxy code itself and permissions so is rather pervasive. The positives are keeping the original source IP and not having to change the workload container to account for the presence of the service mesh.

[hochuenw-dd]

@plwhite thank you for the reply! this really makes sense to me now.