sub-optimal routes on BGP global peer

[murali-reddy]

Thanks @rmb938 @thoro for reporting this issue over gitter.

When nodes in cluster running iBGP advertises routes to the global peer, next hop to subnet is not the node IP corresponding to the subnet. But they all point to single node

For e.g, in 3 node cluster with IP 192.168.1.100, 192.168.1.101, 192.168.1.102 globally peering with 192.168.1.98

root@kube-master:~# gobgp neighbor -u 192.168.1.100
Peer             AS  Up/Down State       |#Received  Accepted
192.168.1.98  64513 00:00:19 Establ      |        3         0
192.168.1.101 64512 00:00:21 Establ      |        1         1
192.168.1.102 64512 00:00:25 Establ      |        1         1
root@kube-master:~# gobgp neighbor -u 192.168.1.102
Peer             AS  Up/Down State       |#Received  Accepted
192.168.1.98  64513 00:00:37 Establ      |        1         0
192.168.1.100 64512 00:00:29 Establ      |        1         1
192.168.1.101 64512 00:00:21 Establ      |        1         1
root@kube-master:~# gobgp neighbor -u 192.168.1.101
Peer             AS  Up/Down State       |#Received  Accepted
192.168.1.98  64513 00:00:37 Establ      |        2         0
192.168.1.100 64512 00:00:26 Establ      |        1         1
192.168.1.102 64512 00:00:22 Establ      |        1         1
root@kube-master:~# gobgp global rib  -u 192.168.1.100
    Network             Next Hop             AS_PATH              Age        Attrs
*>  10.1.0.0/24         192.168.1.100                             00:00:00   [{Origin: i}]
*>  10.1.1.0/24         192.168.1.101                             00:00:47   [{Origin: i} {LocalPref: 100}]
*>  10.1.2.0/24         192.168.1.102                             00:00:51   [{Origin: i} {LocalPref: 100}]
root@kube-master:~# gobgp global rib  -u 192.168.1.101
    Network             Next Hop             AS_PATH              Age        Attrs
*>  10.1.0.0/24         192.168.1.100                             00:00:49   [{Origin: i} {LocalPref: 100}]
*>  10.1.1.0/24         192.168.1.101                             00:00:03   [{Origin: i}]
*>  10.1.2.0/24         192.168.1.102                             00:00:45   [{Origin: i} {LocalPref: 100}]
root@kube-master:~# gobgp global rib  -u 192.168.1.102
    Network             Next Hop             AS_PATH              Age        Attrs
*>  10.1.0.0/24         192.168.1.100                             00:00:54   [{Origin: i} {LocalPref: 100}]
*>  10.1.1.0/24         192.168.1.101                             00:00:46   [{Origin: i} {LocalPref: 100}]
*>  10.1.2.0/24         192.168.1.102                             00:00:05   [{Origin: i}]

routes all point next hop as 192.168.1.102 on BGP global peer

root@router:~# ip route
default via 192.168.1.1 dev ens33 onlink
10.1.0.0/24 via 192.168.1.102 dev ens33  proto zebra
10.1.1.0/24 via 192.168.1.102 dev ens33  proto zebra
10.1.2.0/24 via 192.168.1.102 dev ens33  proto zebra
192.168.1.0/24 dev ens33  proto kernel  scope link  src 192.168.1.98

root@router:~# ip route
default via 192.168.1.1 dev ens33 onlink
10.1.0.0/24 via 192.168.1.102 dev ens33  proto zebra
10.1.1.0/24 via 192.168.1.101 dev ens33  proto zebra
10.1.2.0/24 via 192.168.1.102 dev ens33  proto zebra
192.168.1.0/24 dev ens33  proto kernel  scope link  src 192.168.1.98
root@router:~# ping 10.1.0.70
PING 10.1.0.70 (10.1.0.70) 56(84) bytes of data.
64 bytes from 10.1.0.70: icmp_seq=1 ttl=63 time=2.46 ms
From 192.168.1.102: icmp_seq=2 Redirect Host(New nexthop: 192.168.1.100)
64 bytes from 10.1.0.70: icmp_seq=2 ttl=63 time=0.589 ms
From 192.168.1.102: icmp_seq=3 Redirect Host(New nexthop: 192.168.1.100)
64 bytes from 10.1.0.70: icmp_seq=3 ttl=63 time=0.958 ms
root@router:~# traceroute 10.1.0.70
traceroute to 10.1.0.70 (10.1.0.70), 30 hops max, 60 byte packets
 1  192.168.1.102 (192.168.1.102)  0.307 ms  0.298 ms  0.308 ms
 2  192.168.1.100 (192.168.1.100)  0.648 ms  0.573 ms  0.537 ms
 3  10.1.0.70 (10.1.0.70)  3.238 ms  3.847 ms  3.775 ms

[murali-reddy]

Each node in the cluster advertises with no differenece in attributes(weights/preferences) and with same AS_PATH. So BGP selection randomly picking up next hop.

root@kube-master:~# gobgp global rib  -u 192.168.1.98
      Network             Next Hop             AS_PATH              Age        Attrs
*>  10.1.0.0/24         192.168.1.102        64512                00:00:34   [{Origin: i}]
*   10.1.0.0/24         192.168.1.101        64512                00:00:32   [{Origin: i}]
*   10.1.0.0/24         192.168.1.100        64512                00:00:32   [{Origin: i}]
*>  10.1.1.0/24         192.168.1.102        64512                00:00:34   [{Origin: i}]
*   10.1.1.0/24         192.168.1.101        64512                00:00:32   [{Origin: i}]
*   10.1.1.0/24         192.168.1.100        64512                00:00:32   [{Origin: i}]
*>  10.1.2.0/24         192.168.1.102        64512                00:00:34   [{Origin: i}]
*   10.1.2.0/24         192.168.1.101        64512                00:00:32   [{Origin: i}]
*   10.1.2.0/24         192.168.1.100        64512                00:00:32   [{Origin: i}]

depending on the BGP implementation some attributes are used in selection process

• Weight check
• Local preference check
• Local route check
• AS path length check
  etc

[murali-reddy]

So digging little bit, BGP path attributes can not be used to influence the path selection at ePGP peer. So only logical option seems to use export policy that prevents learned routes from iBGP peers to be advertised to the eBGP global peer.

[murali-reddy]

Fix put in #71 did not quite fix the issue. While ibgp to ebpp peers advertisement works fine. export policy is interfering with routes advertised between iBGP peers.

Re-opening the bug