Egress use external IP on eno2 #6547

yeshl · 2024-07-25T15:32:36Z

node host subnet 192.168.3.0/24 on NIC eno1, external subnet 112.1.6.0/24 on NIC eno2
how to specifying IP 112.1.6.96/24 (SNAT) the traffic from the selected Pods to the external network should use.

apiVersion: crd.antrea.io/v1beta1
kind: ExternalIPPool
metadata:
  name: ip-pool-external
spec:
  ipRanges:
    - start: 112.1.6.96
      end: 112.1.6.111
  subnetInfo:
   gateway: 112.1.6.254
   prefixLength: 24
#  interfaces:  #no this field
#    - eno2
  nodeSelector:
    matchLabels:
      kubernetes.io/hostname: node51
---
apiVersion: crd.antrea.io/v1beta1
kind: Egress
metadata:
  name: egress-external
spec:
  appliedTo:
    namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: dev
    podSelector:
      matchLabels:
        app: busybox
  egressIP: 112.1.6.99
  externalIPPool: ip-pool-external

yeshl · 2024-07-26T08:54:46Z

i want use High-Availability Egress with IPs in subnet 112.1.6.0/24 not in the same subnet as the Node IPs,
as above configuration,EgressIP: 112.1.6.99 on node51,but pods unable to redirected traffic to the node51.
what's wrong with me？

rs20:~# antctl get featuregates|grep Enabled
AntreaPolicy                    Enabled      BETA
AntreaProxy                     Enabled      GA
Egress                          Enabled      BETA
EgressSeparateSubnet            Enabled      ALPHA
EndpointSlice                   Enabled      GA
Multicast                       Enabled      BETA
NetworkPolicyStats              Enabled      BETA
NodeNetworkPolicy               Enabled      ALPHA
NodePortLocal                   Enabled      GA
ServiceExternalIP               Enabled      ALPHA
TopologyAwareHints              Enabled      BETA
Traceflow                       Enabled      BETA

tnqn · 2024-07-26T13:13:17Z

@yeshl Does the node that runs the pod have eno2 and an IP from the 112.1.6.96/24 subnet? Could you collect a support bundle via antctl supportbundle from at least these two nodes and share with us to understand how the node network is configured?

yeshl · 2024-07-26T13:48:41Z

The node50/node51 have tow NIC： eno1(10.0.3.0/24) and eno2(no ip), but other nodes have only one NIC eno1(10.0.3.0/24) . i use metallb High-Availability IP(112.1.6.11/24) on node50/node51,it work very well!! but metallb can only do dnat, i think antrea Egress can do snat，so i use Egress in the same subnet as metallb (112.1.6.11/24)，but it can't work! .....

tnqn · 2024-07-26T15:41:38Z

Is node50 the one running the test Pod? Can other nodes reach eno2's IPs via direct route or default gateway? This is a requirement to make Egress work as mentioned in https://github.com/antrea-io/antrea/blob/main/docs/egress.md#egressip:

The egressIP field specifies the egress (SNAT) IP the traffic from the selected Pods to the external network should use. The IP must be reachable from all Nodes.

antoninbas · 2024-07-26T20:43:43Z

@yeshl is this a good representation of your network topology:

I think you could get it to work if you manually add the routes I wrote in italics in the diagram.
Note that you would need to change your Egress definition to remove the subnetInfo entry altogether.

Edit: I added the onlink keyword by mistake in the route for Node 1. There is no gateway and it is not needed.

@tnqn I feel like we have been getting quite a few queries like this, with similar use cases:

use a different network / interface for Egress
use a dedicated Node for Egress that's connected to the external router (via a separate interface)

I don't know if the current implementation can really handle these use cases very well. There seems to be some general confusion about how subnetInfo can be used - including from me maybe :P

yeshl · 2024-07-27T03:06:54Z

@antoninbas thank u！

#work Node:node31/node32
node31(work pod running):~# ip r add 112.1.6.0/24 dev eno1
node32(work pod running):~# ip r add 112.1.6.0/24 dev eno1

#Egress Node:node50
node50:~# ip r add 112.1.6.0/24 dev eno2 table 100
node50:~# ip r add default via 112.1.6.254 table 100
node50:~# ip ru add from 112.1.6.0/24 lookup 100

For matelLB,As long as the above route is added, it can work very well!!!

antrea egress can work after I add the following route and ip.(but I hope the following route and IP shouldn't be added !，It disrupted high availability for egress)

node50:~# ip ru add from 10.244.8.25 lookup 100  #10.244.8.25 is work pod's IP
node50:~# ip a add 112.1.6.96/24 dev eno2

refer to the implementation of matelLB, it is done very well!

Additionally,The CRD ExternalIPPool appears to use an external IP address such as 112.1.6.99(not 192.168./172./10.*), but in reality it requires the IP in nodes subnet, which is really puzzling!
Is it better to use InternalIPPool or LocalIPPool?

antoninbas · 2024-07-31T18:28:15Z

Additionally,The CRD ExternalIPPool appears to use an external IP address such as 112.1.6.99(not 192.168./172./10.*), but in reality it requires the IP in nodes subnet, which is really puzzling!
Is it better to use InternalIPPool or LocalIPPool?

The Egress IP needs to be routable from any Node, which is not the same as "it requires the IP in nodes subnet"

In the example diagram above, I am adding a route for the Egress "subnet" on every Node manually, which is why the "routable from any Node" requirement is met and why it works.
The EgressSeparateSubnet feature is meant as a mechanism to easily use a different subnet from the Node subnet, without having to program routes manually, but my personal take so far is that it cannot really work with your use case (different interfaces on Egress Node + only the Egress Node is connected to the upstream router).

tnqn · 2024-08-01T16:21:42Z

@tnqn I feel like we have been getting quite a few queries like this, with similar use cases:

use a different network / interface for Egress

use a dedicated Node for Egress that's connected to the external router (via a separate interface)

I don't know if the current implementation can really handle these use cases very well. There seems to be some general confusion about how subnetInfo can be used - including from me maybe :P

@antoninbas Sorry for the late reply. I agree it should be improved or clarified. I will take a look at this and think about it tomorrow or next week.

yeshl · 2024-09-21T03:17:27Z

a flexible solution：

#work nodes:
ip r add 112.1.6.0/24 dev eno1
#egress nodes：
# ip r replace default via 112.1.6.254 dev eno2 onlink table 101 #It can work for a while, but it will be quickly modified by Antrea
ip r add default via 112.1.6.254 dev eno2 onlink table 100
ip ru add fwmark 0x1/0xff lookup 100

tnqn · 2024-09-24T15:27:29Z

@yeshl If I understand your requirement correctly, I think this can be supported by EgressSeparateSubnet, but the only problem prevent it from working without your manual route configuration is that currently the Egress traffic always leave via the "transport interface" even when subnetInfo is provided in the spec of externalIPPool. I tried configuring transportInterface to eno2 and the routes were configured correctly. However, with this configuration nodes that don't have eno2 wouldn't start successfully.

I think this can be resolved by having a separate configuration to specify the interface that the Egress traffic should leave from when subnetInfo is provided (because it could have a different gateway from the node subnet). If I create an image with the necessary code change, would you like to try it? If it can indeed support your use case, we may include the support in next release.

yeshl · 2024-09-26T00:54:18Z

@tnqn, yes i would like to try it!! Actually, we have already used it by add ip rules。but antrea always inserts policy routing at the front！maybe It could specify a priority, just like table 101 priority 101？

apiVersion: crd.antrea.io/v1beta1
kind: ExternalIPPool
metadata:
  name: ip-pool-external
spec:
  ipRanges:
    - start: 112.1.6.96
      end: 112.1.6.111
  subnetInfo:
   gateway: 112.1.6.254
   prefixLength: 24
   transportInterface :  #add this field
     - eno1
  nodeSelector:
    matchLabels:
      kubernetes.io/hostname: node51
---
apiVersion: crd.antrea.io/v1beta1
kind: Egress
metadata:
  name: egress-external
spec:
  appliedTo:
    namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: dev
    podSelector:
      matchLabels:
        app: busybox
  egressIP: 112.1.6.99
  externalIPPool: ip-pool-external
  externalInterface:  eno2  # add this

yeshl · 2024-09-26T01:08:23Z

apiVersion: crd.antrea.io/v1beta1
kind: ExternalIPPool
metadata:
  name: ip-pool-external
spec:
  ipRanges:
    - start: 112.1.6.96
      end: 112.1.6.111
  subnetInfo:
   gateway: 112.1.6.254
   prefixLength: 24
   transportInterface :  #optional，If not specified, use default transport Interface
     - eno1
 ---
apiVersion: crd.antrea.io/v1beta1
kind: Egress
metadata:
  name: egress-external
spec:
  appliedTo:
    namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: dev
    podSelector:
      matchLabels:
        app: busybox
  egressIP: 112.1.6.99
  externalIPPool: ip-pool-external
  externalInterface:  # add this，maybe egress node have different interface name
    -  eno2
    -  eth2
  nodeSelector:  # moving from ExternalIPPool to here seems more reasonable
     matchLabels:
        my.com/node-type: gateway
        kubernetes.io/hostname: node51

tnqn · 2024-09-30T16:20:01Z

@yeshl could you replace the image of antrea-agent with qtian/antrea-agent-ubuntu:v2.2.0-external-interface in your test environment and configure externalInterface as the following example?

apiVersion: v1
kind: ConfigMap
metadata:
  name: antrea-config
  namespace: kube-system
data:
  antrea-agent.conf: |
    externalInterface: eno2

I have tested it in my cluster and it worked. If it also works for you, I will submit a PR.

Note that there are a few reasons why it's in the configuration file but isn't an Egress/ExternalIPPool resource parameter.

K8s API users may not be the cluster admin, and don't have knowledge of the Node network.
It's risky to let API users to specify arbitrary interface name which may mess up the Node network or introduce vulnerabilities.
It's uncommon to have multiple external interfaces. Having to specify the same value in all ExternalIPPools is probably.

This is only a draft to validate the approach. If egress nodes have different interface name, we could make it a list or have some ways to discover the external interface in the official patch.

yeshl · 2024-10-08T03:21:57Z

kubectl get pods -n kube-system -l app=antrea -o=jsonpath='{.items[*].spec.containers[*].image}' | tr ' ' '\n'  
docker.io/qtian/antrea-agent-ubuntu:v2.2.0-external-interface
docker.io/qtian/antrea-agent-ubuntu:v2.2.0-external-interface
docker.io/qtian/antrea-agent-ubuntu:v2.2.0-external-interface
docker.io/qtian/antrea-agent-ubuntu:v2.2.0-external-interface
docker.io/qtian/antrea-agent-ubuntu:v2.2.0-external-interface
docker.io/qtian/antrea-agent-ubuntu:v2.2.0-external-interface
---
 
apiVersion: v1
data:
  antrea-agent.conf: |
    externalInterface: eno4
    # FeatureGates is a map of feature names to bools that enable or disable experimental features.
    featureGates:
    # AllAlpha is a global toggle for alpha features. Per-feature key values override the default set by AllAlpha.
    #  AllAlpha: false
...

---
Antrea Agent Feature Gates
FEATUREGATE                     STATUS       VERSION
AntreaIPAM                      Disabled     ALPHA
AntreaPolicy                    Enabled      BETA
AntreaProxy                     Enabled      GA
BGPPolicy                       Disabled     ALPHA
CleanupStaleUDPSvcConntrack     Enabled      BETA
Egress                          Enabled      BETA
EgressSeparateSubnet            Enabled      ALPHA
EgressTrafficShaping            Disabled     ALPHA
EndpointSlice                   Enabled      GA
ExternalNode                    Disabled     ALPHA
FlowExporter                    Disabled     ALPHA
IPsecCertAuth                   Disabled     ALPHA
L7FlowExporter                  Disabled     ALPHA
L7NetworkPolicy                 Disabled     ALPHA
LoadBalancerModeDSR             Disabled     ALPHA
Multicast                       Enabled      BETA
Multicluster                    Disabled     ALPHA
NetworkPolicyStats              Enabled      BETA
NodeLatencyMonitor              Disabled     ALPHA
NodeNetworkPolicy               Enabled      ALPHA
NodePortLocal                   Enabled      GA
SecondaryNetwork                Disabled     ALPHA
ServiceExternalIP               Enabled      ALPHA
ServiceTrafficDistribution      Enabled      BETA
SupportBundleCollection         Disabled     ALPHA
TopologyAwareHints              Enabled      BETA
Traceflow                       Enabled      BETA
TrafficControl                  Disabled     ALPHA

---
I1008 03:00:07.558838       1 controller.go:397] "Processing ExternalIPPool ADD event" pool="ex-ip-pool-wan" ipRanges=[{"start":"112.1.6.96","end":"112.1.6.111"}]
I1008 03:00:07.558990       1 controller.go:231] "Detected ExternalIPPool event" ExternalIPPool="ex-ip-pool-wan" enqueueServiceNum=0
I1008 03:00:07.565592       1 controller.go:408] "Processing ExternalIPPool UPDATE event" pool="ex-ip-pool-wan" ipRanges=[{"start":"112.1.6.96","end":"112.1.6.111"}]
I1008 03:00:19.706947       1 ip_assigner_linux.go:551] "Assigning IP" ip="112.1.6.97" subnetInfo={"gateway":"112.1.6.254","prefixLength":24} forceAdvertise=true
I1008 03:00:19.707164       1 event.go:389] "Event occurred" object="egress-wan-97" fieldPath="" kind="Egress" apiVersion="crd.antrea.io/v1beta1" type="Normal" reason="IPAssigned" message="Assigned Egress egress-wan-97 with IP 112.1.6.97 on Node node92.host"
I1008 03:00:19.713120       1 controller.go:408] "Processing ExternalIPPool UPDATE event" pool="ex-ip-pool-wan" ipRanges=[{"start":"112.1.6.96","end":"112.1.6.111"}]
E1008 03:00:19.715057       1 egress_controller.go:575] Error syncing Egress egress-wan-97, requeuing. Error: error installing policy route for IP 112.1.6.97: interface for subnet &{112.1.6.254 24 0} not found
I1008 03:00:19.715113       1 ip_assigner_linux.go:551] "Assigning IP" ip="112.1.6.97" subnetInfo={"gateway":"112.1.6.254","prefixLength":24} forceAdvertise=true
E1008 03:00:19.715157       1 egress_controller.go:575] Error syncing Egress egress-wan-97, requeuing. Error: error installing policy route for IP 112.1.6.97: interface for subnet &{112.1.6.254 24 0} not found
I1008 03:00:19.715207       1 event.go:389] "Event occurred" object="egress-wan-97" fieldPath="" kind="Egress" apiVersion="crd.antrea.io/v1beta1" type="Normal" reason="IPAssigned" message="Assigned Egress egress-wan-97 with IP 112.1.6.97 on Node node92.host"
I1008 03:00:24.715662       1 ip_assigner_linux.go:551] "Assigning IP" ip="112.1.6.97" subnetInfo={"gateway":"112.1.6.254","prefixLength":24} forceAdvertise=true
E1008 03:00:24.715748       1 egress_controller.go:575] Error syncing Egress egress-wan-97, requeuing. Error: error installing policy route for IP 112.1.6.97: interface for subnet &{112.1.6.254 24 0} not found
I1008 03:00:24.715855       1 event.go:389] "Event occurred" object="egress-wan-97" fieldPath="" kind="Egress" apiVersion="crd.antrea.io/v1beta1" type="Normal" reason="IPAssigned" message="Assigned Egress egress-wan-97 with IP 112.1.6.97 on Node node92.host"
I1008 03:00:44.716412       1 ip_assigner_linux.go:551] "Assigning IP" ip="112.1.6.97" subnetInfo={"gateway":"112.1.6.254","prefixLength":24} forceAdvertise=true
E1008 03:00:44.716500       1 egress_controller.go:575] Error syncing Egress egress-wan-97, requeuing. Error: error installing policy route for IP 112.1.6.97: interface for subnet &{112.1.6.254 24 0} not found
I1008 03:00:44.716609       1 event.go:389] "Event occurred" object="egress-wan-97" fieldPath="" kind="Egress" apiVersion="crd.antrea.io/v1beta1" type="Normal" reason="IPAssigned" message="Assigned Egress egress-wan-97 with IP 112.1.6.97 on Node node92.host"cer.ingress[].hostname }' 
...

tnqn · 2024-10-08T03:31:01Z

Does Node node92.host has eno4?

yeshl · 2024-10-08T03:33:57Z

when i add a ip on eno4， such as 112.1.6.111/24，it worked！
Subnet should not be used to match network interface,because there is no IP address on the eno4 before,
When eno4 link up, it can work normally！No need to waste a IP on it！

tnqn · 2024-10-08T05:24:51Z

It's kind of expected. The (floating) Egress IPs are assigned to Antrea managed virtual interfaces, instead of the "physical" interfaces, not having any IP on the external interface would break how IP/ARP works on linux, e.g. rp_filter, arp reply. It would also affect how external traffic is routed between other Nodes and this egress Node (you would need to configure extra routes even if the Node is already in the same network). Besides, how would such Nodes reach external network when non Egress IP is assigned to it? Does it make sense rely on VIPs for external connectivity?

For these reasons, the fixed IP on the external interface doesn't seem wasted. And given the complexity of making it work without an fixed IP, it may need to be considered a prerequisite for utilizing the capacity at the moment.

yeshl · 2024-10-08T06:40:27Z

u are right！But our number of IPs is limited, is there a solution that doesn't waste IPs?
eno4 is not the default traffic interface，node‘s default interface is bond0（eno1+eno2）and default routed to a "physical" gateway. We hope to use eno4 as the special ingressIP and egressIP，both with floating IP addresses（Highly available IP address）we use metallb for ingressIP （load balancing IP）it worked very well，use antrea for egressIP！

tnqn · 2024-10-08T08:52:53Z

You mean there are two gateways in the network, one for Node, and one for Pod?

tnqn · 2024-10-08T10:05:47Z

We hope to use eno4 as the special ingressIP and egressIP，both with floating IP addresses（Highly available IP address）we use metallb for ingressIP （load balancing IP）it worked very well，use antrea for egressIP！

You mean metallb for ingressIP can work when there are no fixed IP on eno4? I thought it would have the same issue: fp_filter would drop the packets unless both all and this interface's flag are set to 0.

yeshl · 2024-10-09T01:03:10Z

I hope antrea won't check if there is an IP address on eno4，Just create antrea-egress0 and set the egressIP（floating IP）on it！
it can work well now!
if there is no IP address on the eno4 ，when i reboot node，the interface antrea-egress0 will disappear！
ip_assigner_linux.go:223] "Failed to get external interface" name="eno4" err="unable to find local IP and device"

yes we currently have two gateways, one for default，the other for special (floating) IP .and there will be more in the future, such as connecting to new dedicated lines or VPNs
You can refer to https://blog.aenix.io/configuring-routing-for-metallb-in-l2-mode-7ea26e19219e

tnqn · 2024-10-09T15:37:54Z

My point was that the traffic wouldn't work even if we remove the IP check because how rp_filter and arp work in linux as I mentioned in #6547 (comment). And I think metallb wouldn't work either without fixed IPs on your eno4.

But I see the article you pasted mentioned the same issues that I pointed out and it sounds like you plan to resolve these issues by configuring the nodes' sysctl following the article? If yes, we could remove the IP check in antrea and let users to decide how they make the external interface's routing/arp works. Note that when there is no fixed IP on external interface, you would need to configuring the nodes' sysctl yourself (like configuring all/rp_filter to 0) because the required configurations are not limited to antrea managed interfaces, and generally we don't do such configurations which may be surprising and disruptive to other users, but we could document how to make it work when there is no fixed IP.

yeshl · 2024-10-09T23:54:25Z

@tnqn Yes, you have considered it very thoroughly. I suggest removing the IP check and instead outputting a warning message in the logs, informing the user that they can resolve the issue by setting the IP or using sysctl. This approach is more flexible and gives users more options!

yeshl added the kind/support Categorizes issue or PR as related to a support question. label Jul 25, 2024

This comment was marked as off-topic.

Sign in to view

antoninbas added the area/transit/egress Issues or PRs related to Egress (SNAT for traffic egressing the cluster). label Jul 25, 2024

antoninbas mentioned this issue Sep 3, 2024

K8 Node Transport Interface Detection #6273

Open

tnqn mentioned this issue Oct 9, 2024

Reinitialize ARP/NDP responders when the transport interface changes #6700

Merged

tnqn added this to the Antrea v2.3 release milestone Nov 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Egress use external IP on eno2 #6547

Egress use external IP on eno2 #6547

yeshl commented Jul 25, 2024

This comment was marked as off-topic.

yeshl commented Jul 26, 2024

tnqn commented Jul 26, 2024

yeshl commented Jul 26, 2024

tnqn commented Jul 26, 2024

antoninbas commented Jul 26, 2024 •

edited

Loading

yeshl commented Jul 27, 2024

antoninbas commented Jul 31, 2024

tnqn commented Aug 1, 2024

yeshl commented Sep 21, 2024

tnqn commented Sep 24, 2024

yeshl commented Sep 26, 2024

yeshl commented Sep 26, 2024 •

edited

Loading

tnqn commented Sep 30, 2024 •

edited

Loading

yeshl commented Oct 8, 2024

tnqn commented Oct 8, 2024

yeshl commented Oct 8, 2024 •

edited

Loading

tnqn commented Oct 8, 2024

yeshl commented Oct 8, 2024 •

edited

Loading

tnqn commented Oct 8, 2024

tnqn commented Oct 8, 2024

yeshl commented Oct 9, 2024

tnqn commented Oct 9, 2024

yeshl commented Oct 9, 2024

Egress use external IP on eno2 #6547

Egress use external IP on eno2 #6547

Comments

yeshl commented Jul 25, 2024

This comment was marked as off-topic.

yeshl commented Jul 26, 2024

tnqn commented Jul 26, 2024

yeshl commented Jul 26, 2024

tnqn commented Jul 26, 2024

antoninbas commented Jul 26, 2024 • edited Loading

yeshl commented Jul 27, 2024

antoninbas commented Jul 31, 2024

tnqn commented Aug 1, 2024

yeshl commented Sep 21, 2024

tnqn commented Sep 24, 2024

yeshl commented Sep 26, 2024

yeshl commented Sep 26, 2024 • edited Loading

tnqn commented Sep 30, 2024 • edited Loading

yeshl commented Oct 8, 2024

tnqn commented Oct 8, 2024

yeshl commented Oct 8, 2024 • edited Loading

tnqn commented Oct 8, 2024

yeshl commented Oct 8, 2024 • edited Loading

tnqn commented Oct 8, 2024

tnqn commented Oct 8, 2024

yeshl commented Oct 9, 2024

tnqn commented Oct 9, 2024

yeshl commented Oct 9, 2024

antoninbas commented Jul 26, 2024 •

edited

Loading

yeshl commented Sep 26, 2024 •

edited

Loading

tnqn commented Sep 30, 2024 •

edited

Loading

yeshl commented Oct 8, 2024 •

edited

Loading

yeshl commented Oct 8, 2024 •

edited

Loading