Weave defaults to sleeve mode when encryption is turned off

[paphillon]

What you expected to happen?

Weave should continue to use fastdp when encryption is turned off

What happened?

During some performance run's I noticed pods running on nodes that don't have ingress controller takes twice the amount of time, which indicated there is a network bottleneck. Turning off encryption helped a lot (#3075). However, weave status connections reports it is using sleeve mode when encryption is turned off. Turning on encryption makes it go back to using fastdp.

How to reproduce it?

Turn off encryption
weave status connections

-> 10.51.34.223:6783     established sleeve ea:8b:87:28:29:ce(host1010051034223) mtu=1438
-> 10.51.34.221:6783     established sleeve 4e:78:50:bb:5b:72(host1010051034221) mtu=1438
-> 10.51.34.220:6783     established sleeve 1a:41:f3:2d:a4:eb(host1010051034220) mtu=1438
-> 10.51.34.222:6783     established sleeve c2:ba:89:d8:9b:e9(host1010051034222) mtu=1438
-> 10.51.34.219:6783     failed      cannot connect to ourself, retry: never

turn on encryption

-> 10.51.34.221:6783     established encrypted   fastdp 4e:78:50:bb:5b:72(host1010051034221) encrypted=truemtu=1376
-> 10.51.34.222:6783     established encrypted   fastdp c2:ba:89:d8:9b:e9(host1010051034222) encrypted=truemtu=1376
-> 10.51.34.223:6783     established encrypted   fastdp ea:8b:87:28:29:ce(host1010051034223) encrypted=truemtu=1376
-> 10.51.34.220:6783     established encrypted   fastdp 1a:41:f3:2d:a4:eb(host1010051034220) encrypted=truemtu=1376

Anything else we need to know?

on-prem 5 VM's with CoreOS
4.13.16-coreos-r2 #1 SMP Wed Dec 6 04:27:34 UTC 2017 x86_64 Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz GenuineIntel GNU/Linux

Versions:

$ weave version
weave script 2.1.3
weave 2.1.3
$ docker version
Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.4
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:24:58 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.4
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:24:58 2017
 OS/Arch:      linux/amd64
 Experimental: false

$ uname -a
4.13.16-coreos-r2 #1 SMP Wed Dec 6 04:27:34 UTC 2017 x86_64 Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz GenuineIntel GNU/Linux
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T21:07:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T20:55:30Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

[bboreham]

Can you provide the log file of one of the weave containers please - it should contain clues as to why it chose sleeve.

[paphillon]

@bboreham - I was finally able to get it on fastdp without encryption with a couple of restarts. It seems like a race condition where the container that started first after the change tries to connect to other weave containers that are not yet ready and fall back to sleeve. I noticed the same with encryption turned on too. I will make sure to capture the logs next time I make a switch. In the meantime is there a preferred way to make such configuration changes?
Steps I have been following

Change the deployment configuration 
kubectl apply -f <confg.yaml>
kubectl delete po -n kube-system -l name=weave-net

The above sometimes work, but sometimes does not and finally had to reboot the nodes, which is not an optimal solution.

[rade]

Just applying the new config should usually work. If that fails, I'd delete the daemonset first.