wrong VtepMAC after flannel.1 first up

[zhangguanzhang]

Your Environment

• Flannel version: v0.22.1 mode: vxlan
• k8s version: v1.27.4
• OS version: CentOS Linux release 7.9.2009 (Core)

after node restart

164 info:

$ ip a s flannel.1
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether ee:8d:cf:69:b3:7c brd ff:ff:ff:ff:ff:ff
    inet 172.27.4.0/32 scope global flannel.1

kubernetest annotations:

kubectl get node xxx.164 -o yaml | grep Vtep
    flannel.alpha.coreos.com/backend-data: '{"VNI":1,"VtepMAC":"e6:7c:02:8e:e8:7e"}'

arp table on other nodes, should be ee:8d:cf:69:b3:7c

$ arp -na | grep 172.27.4.0
? (172.27.4.0) at e6:7c:02:8e:e8:7e [ether] PERM on flannel.1

ee:8d:cf:69:b3:7c is not equal e6:7c:02:8e:e8:7e
if I restart the flannel contaienr which running on the node 164, all things will be re-sync and works fine:

$ kubectl get node xxx.164 -o yaml | grep Vtep
    flannel.alpha.coreos.com/backend-data: '{"VNI":1,"VtepMAC":"e6:7c:02:8e:e8:7e"}'
$ kubectl get node xxx.164 -o yaml | grep Vtep
    flannel.alpha.coreos.com/backend-data: '{"VNI":1,"VtepMAC":"e6:7c:02:8e:e8:7e"}'
$ kubectl get node xxx.164 -o yaml | grep Vtep
    flannel.alpha.coreos.com/backend-data: '{"VNI":1,"VtepMAC":"ee:8d:cf:69:b3:7c"}'
$ arp -na | grep 172.27.4.0
? (172.27.4.0) at ee:8d:cf:69:b3:7c [ether] PERM on flannel.1

[zhangguanzhang]

I change the code

flannel/pkg/backend/vxlan/device.go

Lines 124 to 134 in c17e715

	func (dev *vxlanDevice) Configure(ipa ip.IP4Net, flannelnet ip.IP4Net) error {
	if err := ip.EnsureV4AddressOnLink(ipa, flannelnet, dev.link); err != nil {
	return fmt.Errorf("failed to ensure address of interface %s: %s", dev.link.Attrs().Name, err)
	}
	if err := netlink.LinkSetUp(dev.link); err != nil {
	return fmt.Errorf("failed to set interface %s to UP state: %s", dev.link.Attrs().Name, err)
	}
	return nil
	}

to:

func (dev *vxlanDevice) Configure(ipa ip.IP4Net, flannelnet ip.IP4Net) error {
	if err := ip.EnsureV4AddressOnLink(ipa, flannelnet, dev.link); err != nil {
		return fmt.Errorf("failed to ensure address of interface %s: %s", dev.link.Attrs().Name, err)
	}

	log.Infof("before up info:%v", dev.link)

	if err := netlink.LinkSetUp(dev.link); err != nil {
		return fmt.Errorf("failed to set interface %s to UP state: %s", dev.link.Attrs().Name, err)
	}

	nLink, err := netlink.LinkByName(dev.link.LinkAttrs.Name)
	if err == nil {
		if vxlan, ok := nLink.(*netlink.Vxlan); ok {
			log.Infof("after up search vxlan name info:%v", vxlan)
		}
	}

	return nil
}

and I delete flannel.1 and restart flannel container, logs:

I0821 08:30:01.986811       1 device.go:143] before up info:&{{174 1450 0 flannel.1 1a:98:12:11:e2:23 broadcast|multicast 4098 0 0 <nil>  0xc00076a300 0 0 1 <nil> ether <nil> down 0 -1 1 1 65536 65535 [] 0 <nil>} 1 2 1x.xx.xx.164 <nil> 0 0 false false false false false false false false false false false 300 0 8475 0 0}
I0821 08:30:01.987485       1 device.go:152] after up search vxlan name info:&{{174 1450 0 flannel.1 8a:14:6c:ec:93:5c up|broadcast|multicast 69699 0 0 <nil>  0xc00076a000 0 0 1 <nil> ether <nil> unknown 0 -1 1 1 65536 65535 [] 0 <nil>} 1 2 1x.xx.xx.164 <nil> 0 0 false false false false false false false false false false false 300 0 8475 0 0}

seem netlink does't use the mac address

$ kubectl get node xxx.164 -o yaml  | grep Vtep
    flannel.alpha.coreos.com/backend-data: '{"VNI":1,"VtepMAC":"1a:98:12:11:e2:23"}'
$ ip a s flannel.1
174: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether 8a:14:6c:ec:93:5c brd ff:ff:ff:ff:ff:ff
    inet 172.27.4.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::8814:6cff:feec:935c/64 scope link 
       valid_lft forever preferred_lft forever

[zhangguanzhang]

seemd netlink.LinkAdd conflict with file /etc/systemd/network/10-flannel.link:

[Match]
OriginalName=flannel*

[Link]
MACAddressPolicy=random

[rbrtbnfgl]

Is this a default policy on centos?

[zhangguanzhang]

Is this a default policy on centos?

no, I used this file to fix some os which create with clone had same mac address

[rbrtbnfgl]

I think that when you do the LinkUp the mac address is randomized.