Advertising Antrea Egress IP via BGP with FRR-k8s

[adolfomaltez]

Advertising Antrea EgressIP via BGP with FRR-k8s

High Availability on Antrea Egress IP on different L3 subnet advertising via BGP not working as expected.

TL;DR

Is it possible with Antrea version v2.0.0 to achieve Egress HA using a different L3 network segment IP and being dynamically advertised via BGP?

Diagram

I have installed Antrea v2.0.0 and enabled the following features:

• Egress
• ServiceExternalIP
• EgressSeparateSubnet

Installed FRR according to https://github.com/metallb/frr-k8s

Configured FRR to peer with a router external to the cluster, the router is the same gateway for the nodes' network.

frr.yaml.txt

kubectl apply -f frr.yaml

External router configuration:

configure terminal
ip forwarding
router-id 10.63.16.2
router bgp 64500
neighbor 10.63.16.101 remote-as 64500
neighbor 10.63.16.102 remote-as 64500
neighbor 10.63.16.103 remote-as 64500
exit

Deployment with egress no selector and BGP advertisement:

service-egress-no-selector.yaml.txt

kubectl apply -f service-egress-no-selector.yaml

Using Antrea LoadBalancer with an IP (10.10.235.33/32) from a different L3 segment than the nodes, and advertising the IP via BGP using FRR, clients external to the cluster can communicate with the Service LoadBalancer.

External Router BGP routes

The External router shows the IP 10.10.235.33/32 of the LoadBalancer as accessible via any of the 3 worker nodes.

When configuring the Egress 10.10.235.34/32, and advertise the EgressIP via BGP using FRR, the BGP router shows the EgressIP accessible by any of the three worker nodes, which is not accurate since the Egress IP is only accessible through the node that has the EgressIP assigned.
In this way the EgressIP does not work as expected.

The only way I have been able to get Egress to work is by specifying a worker node in EgressIPPool, and advertising the EgressIP via BGP using FRR as reachable only by the node that has the EgressIP.

service-egress-specific-node-selector.yaml.txt

kubectl delete -f service-egress-no-selector.yaml
kubectl apply -f service-egress-specific-node-selector.yaml

External Router BGP routes (with node selector for egress)

Doing this way, the EgressIP works.

But, the HA of the EgressIP is lost, since if the node fails the Egress becomes inaccessible and is not scheduled for another node.

With this scenario, we would have to manually modify the EgressIPPool definition so that it runs the Egress on another node and manually modify the BGP advertisement for that specific new node.

My question is:

Is it possible with Antrea version v2.0.0 to achieve Egress HA using a different L3 network segment IP and being dynamically advertised via BGP?

[tnqn]

@adolfomaltez Thank you for the detailed description. If I understand correctly, the BPG support #5948 which is under actively development may meet your requirement. The API has been merged and the implementation is still under review. I imagine you just need to configure this after it's supported:

apiVersion: crd.antrea.io/v1alpha1
kind: BGPPolicy
metadata:
  name: bgp-egress-ip
spec:
  nodeSelector: {}
  localASN: 64500
  advertisements:
    service:
      ipTypes:
      - LoadBalancerIP
    egress: {}

The current plan is to deliver it in v2.1.0. (v2.0.0 has been released a few months, we don't typically backport new features to patch releases)

[adolfomaltez]

I understand, I will be waiting for the release of version 2.1 and will do the same tests when it is released.
Thank you very much @tnqn

[antoninbas]

I am going to close this as a duplicate of #5948
Antrea native BGP support will make this very easy. When using third-party BGP implementation, you indeed need to ensure that only the Node to which the Egress IP is currently assigned is advertising the IP, which may not always be straightforward for Egress HA (may need to update the configuration dynamically when Egress is re-assigned, if it is not supported out of the box).