Failed to set MTU size >1500 for weave on Red Hat 3.10.0-693.17.1.el7.x86_64

[deshui123]

My project use network-plugin CNI (calico and weave) for kubernetes cluster.
For calico, i can set MTU size >1500
But for weave, the maximum size is 1464.
if size >1464, all weave status will be sleeve, and mtu size checked by weave status connections always 1464
weave version: 2.1.3

How to reproduce:

1. set WEAVE_MTU in /etc/systemd/system/weave.service
   cat /etc/systemd/system/weave.service ..... Environment="CHECKPOINT_DISABLE=1" Environment="WEAVE_MTU=**1500**" .....
2. systemctl daemon-reload
3. systemctl restart weave
4. reboot node
5. ip a or weave status connections

`[root@~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1550 qdisc pfifo_fast state UP qlen 1000
link/ether fa:16:3e:56:77:6d brd ff:ff:ff:ff:ff:ff
inet 192.168.10.45/24 brd 192.168.10.255 scope global dynamic eth0
valid_lft 86332sec preferred_lft 86332sec
inet6 2001:db8:1234::2d/64 scope global dynamic
valid_lft 7134sec preferred_lft 7134sec
inet6 fe80::f816:3eff:fe56:776d/64 scope link
valid_lft forever preferred_lft forever
...
5: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1550 qdisc noqueue state UP

20: datapath: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
link/ether 72:c1:72:da:e3:60 brd ff:ff:ff:ff:ff:ff
inet6 fe80::70c1:72ff:feda:e360/64 scope link
valid_lft forever preferred_lft forever
22: weave: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
link/ether 3e:19:43:00:79:1b brd ff:ff:ff:ff:ff:ff
inet 192.168.1.32/24 scope global weave
valid_lft forever preferred_lft forever
inet6 fe80::3c19:43ff:fe00:791b/64 scope link
valid_lft forever preferred_lft forever
23: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 4e:5b:db:c8:c4:33 brd ff:ff:ff:ff:ff:ff
25: vethwe-datapath@vethwe-bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master datapath state UP
link/ether 1e:97:78:04:bd:2a brd ff:ff:ff:ff:ff:ff
inet6 fe80::1c97:78ff:fe04:bd2a/64 scope link
valid_lft forever preferred_lft forever
26: vethwe-bridge@vethwe-datapath: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master weave state UP
link/ether 6e:7a:b6:e1:c7:ee brd ff:ff:ff:ff:ff:ff
inet6 fe80::6c7a:b6ff:fee1:c7ee/64 scope link
valid_lft forever preferred_lft forever
27: vxlan-6784: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc noqueue master datapath state UNKNOWN qlen 1000
link/ether 8e:33:fc:ab:9c:91 brd ff:ff:ff:ff:ff:ff
inet6 fe80::8c33:fcff:feab:9c91/64 scope link
valid_lft forever preferred_lft forever
29: veth7380096@if28: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1550 qdisc noqueue master docker0 state UP
link/ether ce:12:ad:47:39:cb brd ff:ff:ff:ff:ff:ff link-netnsid 2
inet6 fe80::cc12:adff:fe47:39cb/64 scope link
valid_lft forever preferred_lft forever
31: vethwepl2073d3f@if30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master weave state UP
link/ether f6:80:58:b0:0c:d1 brd ff:ff:ff:ff:ff:ff link-netnsid 3
inet6 fe80::f480:58ff:feb0:cd1/64 scope link
valid_lft forever preferred_lft forever
[root@ ~]$ weave status connections
-> 192.168.10.42:6783 established encrypted sleeve 46:3e:84:d7:5a:75(bcmt-test-control-storage-01) encrypted=truemtu=1464
-> 192.168.10.44:6783 established encrypted sleeve e2:4c:f8:21:2c:6d(bcmt-test-control-storage-03) encrypted=truemtu=1464
-> 192.168.10.47:6783 established encrypted sleeve 12:7a:eb:64:89:a2(bcmt-test-edge-01) encrypted=truemtu=1464
-> 192.168.10.48:6783 established encrypted sleeve 66:d4:a7:67:d0:34(bcmt-test-worker-01) encrypted=truemtu=1464
-> 192.168.10.46:6783 established encrypted sleeve 92:08:53:04:94:16(bcmt-test-edge-02) encrypted=truemtu=1464
-> 192.168.10.49:6783 established encrypted sleeve be:68:f6:4e:02:fd(bcmt-test-worker-03) encrypted=truemtu=1464
-> 192.168.10.43:6783 established encrypted sleeve 7e:15:98:c8:85:c1(bcmt-test-worker-02) encrypted=truemtu=1464
-> 192.168.10.45:6783 failed cannot connect to ourself, retry: never

[root@ ~]$ uname -a
Linux test 3.10.0-693.17.1.el7.x86_64 #1 SMP Sun Jan 14 10:36:03 EST 2018 x86_64 x86_64 x86_64 GNU/Linux`

[bboreham]

Can you clarify how you are starting the Weave daemon? The WEAVE_MTU environment variable is consulted at launch time.

Also it would be useful to provide logs from the weave container.

[deshui123]

logs for WEAVE_MTU is 1500 and 1464

mtu-1464.txt
mtu-1500.txt

One more question:
Does mtu must be divisible by four?

[brb]

@deshui123 Please note, that your public interface (eth0) has MTU of the size 1550 which is probably the max your network/NIC/OS supports.

Weave Net requires MTU to be less than the max because of the encapsulation (VXLAN, ESP). See https://www.weave.works/docs/net/latest/tasks/manage/fastdp/ and https://www.weave.works/blog/weave-net-performance-fast-datapath/ for more details about MTU.

In your case, you have enabled encryption, so to use fastdp the max WEAVE_MTU value must be equal or less than 1550 - 86 = 1464 (yes, the final MTU must be divisible by four).