Skip to content
This repository has been archived by the owner on Jun 20, 2024. It is now read-only.

Weave only allowing connection for one node in cluster in Raspberry Pi 3 #3456

Open
abhinavrawat27 opened this issue Nov 22, 2018 · 0 comments
Open

Weave only allowing connection for one node in cluster in Raspberry Pi 3 #3456

abhinavrawat27 opened this issue Nov 22, 2018 · 0 comments

Comments

@abhinavrawat27
Copy link

What you expected to happen?

Weave should work for all the nodes in the cluster and all the nodes should have proper internet access

What happened?

I have deployed weave in kubernetes cluster. I have 4 nodes running in the cluster. Out of these 4, only 1 gets the proper connection to internet

How to reproduce it?

Install docker : export VERSION=18.06 && curl -sSL get.docker.com | sh
Install kubernetes : sudo apt-get install -qy kubelet=1.9.7-00 kubectl=1.9.7-00 kubeadm=1.9.7-00
Weave: kubectl apply -f https://git.io/weave-kube-1.6
Above installation works fine and the nodes can be added easily. When the daemonset is deployed, only 1 node works fine, rest shows error

Versions:

weave version: 1.6
docker version:

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:30:52 2018
 OS/Arch:           linux/arm
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:26:37 2018
  OS/Arch:          linux/arm
  Experimental:     false

kubectl version:

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.7", GitCommit:"dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92", GitTreeState:"clean", BuildDate:"2018-04-19T00:05:56Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/arm"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.11", GitCommit:"1bfeeb6f212135a22dc787b73e1980e5bccef13d", GitTreeState:"clean", BuildDate:"2018-09-28T21:35:22Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/arm"}

uname -a
Linux TX-G1-209 4.14.82-v7+ #1165 SMP Wed Nov 21 16:42:28 GMT 2018 armv7l GNU/Linux

Logs:

$ kubectl logs -n kube-system weave

DEBU: 2018/11/22 10:20:42.456763 [kube-peers] Checking peer "d6:48:62:21:0e:77" against list &{[{7a:99:6e:57:00:20 tx-g1-209} {d6:48:62:21:0e:77 tx-g1-204}]}
INFO: 2018/11/22 10:20:43.310052 Command line options: map[expect-npc:true nickname:tx-g1-204 docker-api: name:d6:48:62:21:0e:77 port:6783 ipalloc-range:10.32.0.0/12 metrics-addr:0.0.0.0:6782 no-dns:true datapath:datapath db-prefix:/weavedb/weave-net host-root:/host ipalloc-init:consensus=9 conn-limit:100 http-addr:127.0.0.1:6784]
INFO: 2018/11/22 10:20:43.310252 weave  2.5.0
INFO: 2018/11/22 10:20:43.861719 Bridge type is bridged_fastdp
INFO: 2018/11/22 10:20:43.861818 Communication between peers is unencrypted.
INFO: 2018/11/22 10:20:43.894230 Our name is d6:48:62:21:0e:77(tx-g1-204)
INFO: 2018/11/22 10:20:43.894659 Launch detected - using supplied peer list: [192.168.88.172 192.168.88.189 192.168.88.173 192.168.88.80 192.168.88.109 192.168.88.180 192.168.88.73 192.168.88.133 192.168.88.123]
INFO: 2018/11/22 10:20:44.073218 Checking for pre-existing addresses on weave bridge
INFO: 2018/11/22 10:20:44.088128 [allocator d6:48:62:21:0e:77] Initialising with persisted data
INFO: 2018/11/22 10:20:44.109920 Sniffing traffic on datapath (via ODP)
INFO: 2018/11/22 10:20:44.110600 ->[192.168.88.80:6783] attempting connection
INFO: 2018/11/22 10:20:44.111148 ->[192.168.88.180:6783] attempting connection
INFO: 2018/11/22 10:20:44.111490 ->[192.168.88.73:6783] attempting connection
INFO: 2018/11/22 10:20:44.112137 ->[192.168.88.73:39501] connection accepted
INFO: 2018/11/22 10:20:44.112373 ->[192.168.88.109:6783] attempting connection
INFO: 2018/11/22 10:20:44.112739 ->[192.168.88.173:6783] attempting connection
INFO: 2018/11/22 10:20:44.113041 ->[192.168.88.133:6783] attempting connection
INFO: 2018/11/22 10:20:44.113323 ->[192.168.88.123:6783] attempting connection
INFO: 2018/11/22 10:20:44.113707 ->[192.168.88.189:6783] attempting connection
INFO: 2018/11/22 10:20:44.113976 ->[192.168.88.172:6783] attempting connection
INFO: 2018/11/22 10:20:44.115007 ->[192.168.88.73:39501|d6:48:62:21:0e:77(tx-g1-204)]: connection shutting down due to error: cannot connect to ourself
INFO: 2018/11/22 10:20:44.115954 ->[192.168.88.73:6783|d6:48:62:21:0e:77(tx-g1-204)]: connection shutting down due to error: cannot connect to ourself
INFO: 2018/11/22 10:20:44.158061 Listening for HTTP control messages on 127.0.0.1:6784
INFO: 2018/11/22 10:20:44.159223 Listening for metrics requests on 0.0.0.0:6782
INFO: 2018/11/22 10:20:44.244145 ->[192.168.88.80:6783|d6:48:62:21:0e:77(tx-g1-204)]: connection shutting down due to error: local "d6:48:62:21:0e:77(tx-g1-204)" and remote "d6:48:62:21:0e:77(tx-g1-192)" peer names collision
INFO: 2018/11/22 10:20:44.251446 ->[192.168.88.173:6783|d6:48:62:21:0e:77(tx-g1-204)]: connection shutting down due to error: local "d6:48:62:21:0e:77(tx-g1-204)" and remote "d6:48:62:21:0e:77(tx-g1-187)" peer names collision
INFO: 2018/11/22 10:20:44.259880 ->[192.168.88.172:6783|d6:48:62:21:0e:77(tx-g1-204)]: connection shutting down due to error: local "d6:48:62:21:0e:77(tx-g1-204)" and remote "d6:48:62:21:0e:77(tx-g1-169)" peer names collision
INFO: 2018/11/22 10:20:44.261005 ->[192.168.88.189:6783|d6:48:62:21:0e:77(tx-g1-204)]: connection shutting down due to error: local "d6:48:62:21:0e:77(tx-g1-204)" and remote "d6:48:62:21:0e:77(tx-g1-172)" peer names collision
INFO: 2018/11/22 10:20:44.262175 ->[192.168.88.133:6783|d6:48:62:21:0e:77(tx-g1-204)]: connection shutting down due to error: local "d6:48:62:21:0e:77(tx-g1-204)" and remote "d6:48:62:21:0e:77(tx-g1-206)" peer names collision
INFO: 2018/11/22 10:20:44.339492 ->[192.168.88.123:6783|7a:99:6e:57:00:20(tx-g1-209)]: connection ready; using protocol version 2
INFO: 2018/11/22 10:20:44.339853 overlay_switch ->[7a:99:6e:57:00:20(tx-g1-209)] using fastdp
INFO: 2018/11/22 10:20:44.340027 ->[192.168.88.123:6783|7a:99:6e:57:00:20(tx-g1-209)]: connection added (new peer)
INFO: 2018/11/22 10:20:44.345293 ->[192.168.88.123:6783|7a:99:6e:57:00:20(tx-g1-209)]: connection shutting down due to error: read tcp4 192.168.88.73:41475->192.168.88.123:6783: read: connection reset by peer
INFO: 2018/11/22 10:20:44.345560 ->[192.168.88.123:6783|7a:99:6e:57:00:20(tx-g1-209)]: connection deleted
INFO: 2018/11/22 10:20:44.345670 Removed unreachable peer 7a:99:6e:57:00:20(tx-g1-209)
INFO: 2018/11/22 10:20:44.993330 [kube-peers] Added myself to peer list &{[{7a:99:6e:57:00:20 tx-g1-209} {d6:48:62:21:0e:77 tx-g1-204}]}
DEBU: 2018/11/22 10:20:45.064310 [kube-peers] Nodes that have disappeared: map[]
10.36.0.0
192.168.88.172
192.168.88.189
192.168.88.173
192.168.88.80
192.168.88.109
192.168.88.180
192.168.88.73
192.168.88.133
192.168.88.123
DEBU: 2018/11/22 10:20:45.935336 registering for updates for node delete events
INFO: 2018/11/22 10:20:46.400945 ->[192.168.88.123:6783] attempting connection
INFO: 2018/11/22 10:20:46.421560 ->[192.168.88.123:6783|7a:99:6e:57:00:20(tx-g1-209)]: connection ready; using protocol version 2
INFO: 2018/11/22 10:20:46.421857 overlay_switch ->[7a:99:6e:57:00:20(tx-g1-209)] using fastdp
INFO: 2018/11/22 10:20:46.421991 ->[192.168.88.123:6783|7a:99:6e:57:00:20(tx-g1-209)]: connection added (new peer)
INFO: 2018/11/22 10:20:46.424036 ->[192.168.88.123:6783|7a:99:6e:57:00:20(tx-g1-209)]: connection shutting down due to error: write tcp4 192.168.88.73:57443->192.168.88.123:6783: write: connection reset by peer
INFO: 2018/11/22 10:20:46.424382 ->[192.168.88.123:6783|7a:99:6e:57:00:20(tx-g1-209)]: connection deleted
INFO: 2018/11/22 10:20:46.424513 Removed unreachable peer 7a:99:6e:57:00:20(tx-g1-209)
INFO: 2018/11/22 10:20:47.261932 ->[192.168.88.180:6783] error during connection attempt: dial tcp4 :0->192.168.88.180:6783: connect: no route to host
INFO: 2018/11/22 10:20:48.354767 ->[192.168.88.180:6783] attempting connection
INFO: 2018/11/22 10:20:48.982527 ->[192.168.88.123:6783] attempting connection
INFO: 2018/11/22 10:20:48.992886 ->[192.168.88.123:6783|7a:99:6e:57:00:20(tx-g1-209)]: connection ready; using protocol version 2
INFO: 2018/11/22 10:20:48.993134 overlay_switch ->[7a:99:6e:57:00:20(tx-g1-209)] using fastdp
INFO: 2018/11/22 10:20:48.993354 ->[192.168.88.123:6783|7a:99:6e:57:00:20(tx-g1-209)]: connection added (new peer)
INFO: 2018/11/22 10:20:48.997918 ->[192.168.88.123:6783|7a:99:6e:57:00:20(tx-g1-209)]: connection shutting down due to error: read tcp4 192.168.88.73:59151->192.168.88.123:6783: read: connection reset by peer
INFO: 2018/11/22 10:20:48.998266 ->[192.168.88.123:6783|7a:99:6e:57:00:20(tx-g1-209)]: connection deleted
INFO: 2018/11/22 10:20:48.998439 Removed unreachable peer 7a:99:6e:57:00:20(tx-g1-209)
INFO: 2018/11/22 10:20:50.372397 ->[192.168.88.180:6783] error during connection attempt: dial tcp4 :0->192.168.88.180:6783: connect: no route to host
INFO: 2018/11/

Anything else we need to know?

Using Raspberry pi 3b+. While looking for this issue solution, I found this link #3310 (comment) which says deleting the file /var/lib/weave/weave-netdata.db. It did resolved the issue after a reboot but the same issue occurs after sometime. As per few comments in this post #3314 I have also upgraded the RPBi kernel and then installed the kube and weave, but the issue remains the same, The same post also discussed about disaling the fastdb but looks like I am not able to disable fastdb. Also I do not want this as I have read its a bit slow and consumes more usage.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@abhinavrawat27