Add Antrea based traffic redirection in Istio CNI plugin when using Ambient mode

[tschwaller]

The new Istio Ambient mode redirects L4 traffic to the ztunnel pods on the nodes using

• iptables and Geneve tunnels or
• eBPF programs and maps

The ztunnel pod handles mTLS connections to ztunnel pods on other nodes. You can also configure simple firewall rules
using Istio AuhorizationPolicies. The traffic redirection is using the Istio CNI plugin in CNI chaining mode.

The traffic redirection should be possible and more efficient with the Antrea TrafficControl CRD (instead of iptables and Geneve tunnels) but needs further analysis from the Antrea team.

The suggestion is to contribute a new Antrea redirection mode in the Istio CNI plugin or to work with the Istio Ambient team to have a mode which does not need the Istio CNI plugin at all in an Antrea context (which would then not support the (legacy) sidecar based approach which the Istio CNI plugin also handles but only the new Ambient mode).

One should also keep in mind that before the traffic reaches the ztunnel pod it might have to be analyzed by the Suricata engine on the node if this feature is used together with Antrea (pod -> Suricata -> ztunnel -> ...).

The Istio L4 AuthorizationPolicies might also be enhanced using Antrea (Cluster ) Network Policies or the new AdminNetworkPolicies, but this needs further analysis.

Last but not least one has to consider some overlap with the Antrea Multi-Cluster implementation which is using a very similar architecture with the Multi-Cluster Gateway Pods (but without mTLS). This looks more like an upstream discussion about which specification/project handles which layer in the networking stack.

[antoninbas]

Given that the ztunnel Pod runs as a regular Pod (not hostNetwork), it should be pretty easy for us to redirect mesh traffic to it directly in OVS.

It would be great if we could use TrafficControl for this, with a single CR for all the Pods which are part of the mesh. However, at the moment with TrafficControl, you need to define a destination (target port), which can be 1) a device name, or 2) a VTEP IP. For the Ambient mesh case, both the Pod's device name and the Pod's IP will vary for each ztunnel Pod (i.e., for each Node). So if we want to be able to use the TrafficControl CRD, and define a single resource for the service mesh, we may need a more flexible way of selecting the target and return ports (which I assume would be the same here).

1. We could do a quick validation where we encap packets in OVS with Geneve, and send them directly to the ztunnel Pod, instead of having the packets go through the host network and iptables on the host. We also need to intercept and decap packets in the other direction. This would require no change in Istio AFAIK.

2. We should also be able to send the packets to the ztunnel Pod directly, without modification or encapsulation. Inside the ztunnel Pod, we would keep using TPROXY (either with iptables or eBPF?), which is what Istio does today. This would probably require some changes on the Istio side (ztunnel network configuration).

Adding @tnqn who is the TrafficControl expert.

Some useful links:

• https://tetrate.io/blog/using-geneve-tunnels-to-implement-istio-ambient-mesh-traffic-interception/
• https://tetrate.io/blog/transparent-traffic-intercepting-and-routing-in-the-l4-network-of-istio-ambient-mesh/
• https://www.solo.io/blog/traffic-ambient-mesh-ztunnel-ebpf-waypoint/

[tnqn]

@tschwaller @antoninbas I have tried Istio Ambient mode with Antrea and managed to make them work together, with a number of workarounds. Here are my findings:

For initialization, I found the current implementation of Ambient mode has an assumption on Pod network's implementation: It assumes the Pod is connected to host network via veth and there is a route entry for the Pod IP, which will be used to identify the ztunnel's veth and then the network namespace of ztunnel Pod. As Antrea doesn't have such per Pod route in host network, the istio-cni-node Pod failed to initialize some network resources. I worked around it by changing istio-cni-node to look for interface by veth name. This shouldn't be a problem as it's not hard to figure out a more generic way to identify Pod's interface.

Then I tried to use TrafficControl to redirect some Pods' traffic to ztunnel Pod via geneve tunnel. However, it can't work automatically because of the assumption on Pod network's implementation. After I resolved some rp_filter, policy routing, tunnel source IP mismatching issues, the encapsulated request was finally delivered to client side ztunnel Pod and triggered a response (SYN+ACK). However, the response didn't go symmetric path and didn't get encapsulated like the request, it seems assuming the response could be delivered to client automatically. But it would be dropped by Antrea's SpoofGuard flow. I didn't go this route further considering it looks more complicated.

The second attemp is to use TrafficControl to redirect Pod traffic to ztunnel Pod's port without encapsulation. As Antonin mentioned, the current API doesn't support specifying different receivers on different Nodes. I worked around it by creating two TrafficControl resources, one for client and one for server. The traffic is redirected to ztunnel Pod expectedly without encapsulation. Then to make ztunnel accept the traffic, I made a little change to TrafficControl flows to overwrite the destination MAC to the receiver Pod's MAC address so it won't be dropped when checking MAC. The last problem is, currently ztunnel differentiates redirected traffic by device name (redirected traffic is expected to be received on the geneve port pistioout), I made a little change to make it accept traffic by source IP (we could also use pkt_mark). After resolving these problems on both client and server Nodes side, the access succeeded:

# kubectl exec deploy/notsleep -- curl -s http://10.96.73.75:9080/ | grep -o "<title>.*</title>"
<title>Simple Bookstore App</title>

ztunnel on client side:

2023-11-09T11:03:35.965904Z  INFO outbound{id=3439f9fd7b82f709128abe6e98b3af54}: ztunnel::proxy::outbound: Proxying to 10.244.1.10:9080 using TCP via 10.244.1.10:9080 type Direct
2023-11-09T11:03:35.976537Z  INFO outbound{id=3439f9fd7b82f709128abe6e98b3af54}: ztunnel::proxy::outbound: complete dur=10.837747ms

ztunnel on server side:

2023-11-09T11:03:35.969110Z  INFO ztunnel::proxy::inbound_passthrough: accepted connection source=10.244.2.7:38493 destination=10.244.1.10:9080 component="inbound plaintext"
2023-11-09T11:03:35.976494Z  INFO ztunnel::proxy::inbound_passthrough: connection complete source=10.244.2.7:38493 destination=10.244.1.10:9080 component="inbound plaintext"

So I think redirecting traffic to ztunnel without encapsulation is promising. It should be simpler and more efficient. To make the whole workflow work automatically, changes are needed on both Istio and Antrea side, but there should be generic ways to make these changes.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days. You can add a label "lifecycle/frozen" to skip stale checking.