Skip to content
This repository has been archived by the owner on Jun 20, 2024. It is now read-only.

masters fight over the same range #3721

Closed
kostyrev opened this issue Oct 17, 2019 · 5 comments
Closed

masters fight over the same range #3721

kostyrev opened this issue Oct 17, 2019 · 5 comments

Comments

@kostyrev
Copy link 

> kubectl get nodes -l  kubernetes.io/role=master -owide
NAME                             STATUS   ROLES    AGE   VERSION    INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
ip-10-128-203-252.ec2.internal   Ready    master   84m   v1.13.10   10.128.203.252   <none>        Ubuntu 16.04.6 LTS   4.4.0-1087-aws   docker://18.6.3
ip-10-128-223-78.ec2.internal    Ready    master   94m   v1.13.10   10.128.223.78    <none>        Ubuntu 16.04.6 LTS   4.4.0-1087-aws   docker://18.6.3
ip-10-128-234-57.ec2.internal    Ready    master   76m   v1.13.10   10.128.234.57    <none>        Ubuntu 16.04.6 LTS   4.4.0-1087-aws   docker://18.6.3

> k version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T16:23:09Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.10", GitCommit:"37d169313237cb4ceb2cc4bef300f2ae3053c1a2", GitTreeState:"clean", BuildDate:"2019-08-19T10:44:49Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}

INFO: 2019/10/17 16:25:52.322198 ->[10.128.231.9:47637|ae:10:6f:36:96:ef(ip-10-128-231-9.ec2.internal)]: connection shutting down due to error: Inconsistent entries for 10.32.0.128: owned by b2:f8:2e:00:01:60 but incoming message says 0e:d4:f3:53:1c:91

b2:f8:2e:00:01:60(ip-10-128-203-252.ec2.internal)   390166 IPs (37.2% of total) (3 active)
0e:d4:f3:53:1c:91(ip-10-128-234-57.ec2.internal)   354868 IPs (33.8% of total) (2 active)
de:de:81:69:3b:68(ip-10-128-223-78.ec2.internal)

master-us-east-1a ip-10-128-203-252.ec2.internal
master-us-east-1b ip-10-128-223-78.ec2.internal
master-us-east-1c ip-10-128-234-57.ec2.internal

weave --local status

weave-net-pj8pg

        Version: 2.5.2 (up to date; next check at 2019/10/17 22:27:11)
        Service: router
       Protocol: weave 1..2
           Name: b2:f8:2e:00:01:60(ip-10-128-203-252.ec2.internal)
     Encryption: disabled
  PeerDiscovery: enabled
        Targets: 53
    Connections: 54 (53 established, 1 failed)
          Peers: 54 (with 2854 established, 8 pending connections)
 TrustedSubnets: none

        Service: ipam
         Status: ready
          Range: 10.32.0.0/12
  DefaultSubnet: 10.32.0.0/12

weave-net-24nkq

        Version: 2.5.2 (up to date; next check at 2019/10/17 22:58:03)
        Service: router
       Protocol: weave 1..2
           Name: de:de:81:69:3b:68(ip-10-128-223-78.ec2.internal)
     Encryption: disabled
  PeerDiscovery: enabled
        Targets: 53
    Connections: 54 (53 established, 1 failed)
          Peers: 54 (with 2856 established, 6 pending connections)
 TrustedSubnets: none

        Service: ipam
         Status: ready
          Range: 10.32.0.0/12
  DefaultSubnet: 10.32.0.0/12

weave-net-hd9dv

        Version: 2.5.2 (up to date; next check at 2019/10/17 22:41:53)
        Service: router
       Protocol: weave 1..2
           Name: 0e:d4:f3:53:1c:91(ip-10-128-234-57.ec2.internal)
     Encryption: disabled
  PeerDiscovery: enabled
        Targets: 53
    Connections: 54 (53 established, 1 failed)
          Peers: 54 (with 2853 established, 8 pending connections)
 TrustedSubnets: none

        Service: ipam
         Status: ready
          Range: 10.32.0.0/12
  DefaultSubnet: 10.32.0.0/12

weave --local status ipam;done

weave-net-pj8pg

b2:f8:2e:00:01:60(ip-10-128-203-252.ec2.internal)   394270 IPs (37.6% of total) (3 active)
86:bc:eb:83:71:c4(ip-10-128-204-234.ec2.internal)     8065 IPs (00.8% of total) 
7a:9c:fe:53:31:2c(ip-10-128-208-168.ec2.internal)     4102 IPs (00.4% of total) 
d6:46:38:d6:2c:0e(ip-10-128-200-90.ec2.internal)     8192 IPs (00.8% of total) 
9e:f2:e0:3b:a3:fb(ip-10-128-221-77.ec2.internal)       56 IPs (00.0% of total) 
42:9f:5b:98:56:2f(ip-10-128-230-233.ec2.internal)     8206 IPs (00.8% of total) 
3e:65:d5:e4:85:e8(ip-10-128-226-205.ec2.internal)     4102 IPs (00.4% of total) 
56:de:40:12:bb:8d(ip-10-128-225-148.ec2.internal)    20485 IPs (02.0% of total) 
e2:f3:fa:50:cd:67(ip-10-128-239-45.ec2.internal)     4101 IPs (00.4% of total) 
32:bb:84:80:35:a4(ip-10-128-235-176.ec2.internal)     8192 IPs (00.8% of total) 
be:38:56:4d:00:02(ip-10-128-193-210.ec2.internal)     8736 IPs (00.8% of total) 
5e:58:a6:d8:43:31(ip-10-128-206-176.ec2.internal)    40967 IPs (03.9% of total) 
1a:84:8d:a9:c4:17(ip-10-128-198-83.ec2.internal)     1589 IPs (00.2% of total) 
e2:3d:21:ef:02:5f(ip-10-128-217-187.ec2.internal)     5799 IPs (00.6% of total) 
8a:8b:d1:17:f7:33(ip-10-128-219-144.ec2.internal)    32777 IPs (03.1% of total) 
0a:55:01:96:29:9d(ip-10-128-235-0.ec2.internal)     8240 IPs (00.8% of total) 
9e:d3:58:e6:39:30(ip-10-128-205-86.ec2.internal)     2048 IPs (00.2% of total) 
1a:6e:d5:21:cb:5d(ip-10-128-208-254.ec2.internal)      768 IPs (00.1% of total) 
9a:5e:ca:97:75:1e(ip-10-128-197-65.ec2.internal)    11529 IPs (01.1% of total) 
ae:8e:b3:e1:43:2f(ip-10-128-222-176.ec2.internal)     2059 IPs (00.2% of total) 
6a:f0:40:bb:b4:ce(ip-10-128-209-244.ec2.internal)     8192 IPs (00.8% of total) 
72:a2:ef:9f:11:d6(ip-10-128-197-174.ec2.internal)       15 IPs (00.0% of total) 
12:65:19:d1:38:db(ip-10-128-200-147.ec2.internal)    59330 IPs (05.7% of total) 
26:f5:1a:4e:36:68(ip-10-128-225-55.ec2.internal)       11 IPs (00.0% of total) 
ae:61:cd:c6:62:19(ip-10-128-211-129.ec2.internal)     2048 IPs (00.2% of total) 
de:f8:54:f2:0c:28(ip-10-128-234-112.ec2.internal)    12288 IPs (01.2% of total) 
9e:62:8c:9e:97:69(ip-10-128-224-168.ec2.internal)    16384 IPs (01.6% of total) 
82:29:dd:ef:df:75(ip-10-128-230-179.ec2.internal)     4096 IPs (00.4% of total) 
06:b3:f3:20:bc:be(ip-10-128-192-116.ec2.internal)    22528 IPs (02.1% of total) 
0a:9e:be:0d:d7:e4(ip-10-128-205-71.ec2.internal)     4107 IPs (00.4% of total) 
0a:f2:cb:be:ff:8e(ip-10-128-233-236.ec2.internal)     7865 IPs (00.8% of total) 
5a:57:4a:3a:e6:5d(ip-10-128-198-150.ec2.internal)    15176 IPs (01.4% of total) 
1e:2b:8e:70:3e:f0(ip-10-128-216-176.ec2.internal)     1032 IPs (00.1% of total) 
16:6b:0d:0f:fc:90(ip-10-128-222-185.ec2.internal)       64 IPs (00.0% of total) 
7a:07:9a:90:f6:4d(ip-10-128-213-28.ec2.internal)     4385 IPs (00.4% of total) 
66:70:40:ae:e9:bd(ip-10-128-193-152.ec2.internal)    12288 IPs (01.2% of total) 
42:73:b4:55:6e:9d(ip-10-128-214-51.ec2.internal)    26635 IPs (02.5% of total) 
42:06:78:d0:55:9a(ip-10-128-208-59.ec2.internal)    12288 IPs (01.2% of total) 
56:1d:12:15:66:5b(ip-10-128-196-192.ec2.internal)     4198 IPs (00.4% of total) 
92:2a:90:53:2a:04(ip-10-128-207-7.ec2.internal)      384 IPs (00.0% of total) 
7e:1f:ef:1d:26:86(ip-10-128-236-11.ec2.internal)    16391 IPs (01.6% of total) 
4a:83:c5:7a:e0:53(ip-10-128-201-80.ec2.internal)    24582 IPs (02.3% of total) 
86:05:95:ef:89:5c(ip-10-128-235-185.ec2.internal)     8192 IPs (00.8% of total) 
0e:d4:f3:53:1c:91(ip-10-128-234-57.ec2.internal)     2048 IPs (00.2% of total) 
b2:e7:0e:fd:16:3c(ip-10-128-210-128.ec2.internal)     8192 IPs (00.8% of total) 
6e:c8:93:2b:1b:8c(ip-10-128-223-143.ec2.internal)     9114 IPs (00.9% of total) 
96:7d:75:05:d9:db(ip-10-128-221-184.ec2.internal)   116581 IPs (11.1% of total) 
06:79:13:37:4e:9c(ip-10-128-204-95.ec2.internal)    10331 IPs (01.0% of total) 
36:1f:70:99:66:63(ip-10-128-231-254.ec2.internal)    65548 IPs (06.3% of total)

weave-net-24nkq

de:de:81:69:3b:68(ip-10-128-223-78.ec2.internal)   344504 IPs (32.9% of total) (3 active)
1e:2b:8e:70:3e:f0(ip-10-128-216-176.ec2.internal)    34217 IPs (03.3% of total) 
36:1f:70:99:66:63(ip-10-128-231-254.ec2.internal)    70821 IPs (06.8% of total) 
5e:58:a6:d8:43:31(ip-10-128-206-176.ec2.internal)    28040 IPs (02.7% of total) 
42:9f:5b:98:56:2f(ip-10-128-230-233.ec2.internal)    24583 IPs (02.3% of total) 
4a:83:c5:7a:e0:53(ip-10-128-201-80.ec2.internal)    12288 IPs (01.2% of total) 
56:1d:12:15:66:5b(ip-10-128-196-192.ec2.internal)      296 IPs (00.0% of total) 
26:f5:1a:4e:36:68(ip-10-128-225-55.ec2.internal)    16384 IPs (01.6% of total) 
7a:07:9a:90:f6:4d(ip-10-128-213-28.ec2.internal)     4385 IPs (00.4% of total) 
9e:f2:e0:3b:a3:fb(ip-10-128-221-77.ec2.internal)    12288 IPs (01.2% of total) 
be:38:56:4d:00:02(ip-10-128-193-210.ec2.internal)       11 IPs (00.0% of total) 
ae:61:cd:c6:62:19(ip-10-128-211-129.ec2.internal)        4 IPs (00.0% of total) 
0a:9e:be:0d:d7:e4(ip-10-128-205-71.ec2.internal)    12288 IPs (01.2% of total) 
d6:46:38:d6:2c:0e(ip-10-128-200-90.ec2.internal)       64 IPs (00.0% of total) 
56:de:40:12:bb:8d(ip-10-128-225-148.ec2.internal)    28215 IPs (02.7% of total) 
12:65:19:d1:38:db(ip-10-128-200-147.ec2.internal)     4107 IPs (00.4% of total) 
1a:84:8d:a9:c4:17(ip-10-128-198-83.ec2.internal)     4105 IPs (00.4% of total) 
06:79:13:37:4e:9c(ip-10-128-204-95.ec2.internal)     8218 IPs (00.8% of total) 
82:29:dd:ef:df:75(ip-10-128-230-179.ec2.internal)       12 IPs (00.0% of total) 
9a:5e:ca:97:75:1e(ip-10-128-197-65.ec2.internal)   179054 IPs (17.1% of total) 
ae:8e:b3:e1:43:2f(ip-10-128-222-176.ec2.internal)    12664 IPs (01.2% of total) 
e2:f3:fa:50:cd:67(ip-10-128-239-45.ec2.internal)    35615 IPs (03.4% of total) 
0a:f2:cb:be:ff:8e(ip-10-128-233-236.ec2.internal)       34 IPs (00.0% of total) 
a6:ec:c3:0a:97:ef(ip-10-128-196-11.ec2.internal)     8192 IPs (00.8% of total) 
5a:57:4a:3a:e6:5d(ip-10-128-198-150.ec2.internal)    21517 IPs (02.1% of total) 
66:70:40:ae:e9:bd(ip-10-128-193-152.ec2.internal)       10 IPs (00.0% of total) 
16:6b:0d:0f:fc:90(ip-10-128-222-185.ec2.internal)       64 IPs (00.0% of total) 
42:73:b4:55:6e:9d(ip-10-128-214-51.ec2.internal)    12288 IPs (01.2% of total) 
9e:d3:58:e6:39:30(ip-10-128-205-86.ec2.internal)       48 IPs (00.0% of total) 
92:2a:90:53:2a:04(ip-10-128-207-7.ec2.internal)      384 IPs (00.0% of total) 
7a:9c:fe:53:31:2c(ip-10-128-208-168.ec2.internal)    34764 IPs (03.3% of total) 
8a:8b:d1:17:f7:33(ip-10-128-219-144.ec2.internal)    17399 IPs (01.7% of total) 
e2:3d:21:ef:02:5f(ip-10-128-217-187.ec2.internal)       10 IPs (00.0% of total) 
de:f8:54:f2:0c:28(ip-10-128-234-112.ec2.internal)     8199 IPs (00.8% of total) 
0a:55:01:96:29:9d(ip-10-128-235-0.ec2.internal)       10 IPs (00.0% of total) 
36:75:46:6d:75:50(ip-10-128-200-85.ec2.internal)    10368 IPs (01.0% of total) 
86:05:95:ef:89:5c(ip-10-128-235-185.ec2.internal)      768 IPs (00.1% of total) 
3e:65:d5:e4:85:e8(ip-10-128-226-205.ec2.internal)    12554 IPs (01.2% of total) 
72:a2:ef:9f:11:d6(ip-10-128-197-174.ec2.internal)       15 IPs (00.0% of total) 
6e:c8:93:2b:1b:8c(ip-10-128-223-143.ec2.internal)    37644 IPs (03.6% of total) 
86:bc:eb:83:71:c4(ip-10-128-204-234.ec2.internal)     8061 IPs (00.8% of total) 
66:d9:02:67:b0:3e(ip-10-128-221-211.ec2.internal)     8192 IPs (00.8% of total) 
7e:1f:ef:1d:26:86(ip-10-128-236-11.ec2.internal)    14338 IPs (01.4% of total) 
96:7d:75:05:d9:db(ip-10-128-221-184.ec2.internal)    12295 IPs (01.2% of total) 
42:06:78:d0:55:9a(ip-10-128-208-59.ec2.internal)     8203 IPs (00.8% of total) 
b2:e7:0e:fd:16:3c(ip-10-128-210-128.ec2.internal)       32 IPs (00.0% of total) 
6a:f0:40:bb:b4:ce(ip-10-128-209-244.ec2.internal)     1024 IPs (00.1% of total)

weave-net-hd9dv

0e:d4:f3:53:1c:91(ip-10-128-234-57.ec2.internal)   354868 IPs (33.8% of total) (2 active)
e2:f3:fa:50:cd:67(ip-10-128-239-45.ec2.internal)    35615 IPs (03.4% of total) 
e2:3d:21:ef:02:5f(ip-10-128-217-187.ec2.internal)    12298 IPs (01.2% of total) 
26:f5:1a:4e:36:68(ip-10-128-225-55.ec2.internal)    16384 IPs (01.6% of total) 
82:29:dd:ef:df:75(ip-10-128-230-179.ec2.internal)       12 IPs (00.0% of total) 
5e:58:a6:d8:43:31(ip-10-128-206-176.ec2.internal)    28040 IPs (02.7% of total) 
d6:46:38:d6:2c:0e(ip-10-128-200-90.ec2.internal)       64 IPs (00.0% of total) 
4a:83:c5:7a:e0:53(ip-10-128-201-80.ec2.internal)    12288 IPs (01.2% of total) 
9e:d3:58:e6:39:30(ip-10-128-205-86.ec2.internal)       48 IPs (00.0% of total) 
6e:c8:93:2b:1b:8c(ip-10-128-223-143.ec2.internal)    29452 IPs (02.8% of total) 
de:f8:54:f2:0c:28(ip-10-128-234-112.ec2.internal)     8199 IPs (00.8% of total) 
be:38:56:4d:00:02(ip-10-128-193-210.ec2.internal)    12299 IPs (01.2% of total) 
42:73:b4:55:6e:9d(ip-10-128-214-51.ec2.internal)    12288 IPs (01.2% of total) 
6a:f0:40:bb:b4:ce(ip-10-128-209-244.ec2.internal)     1024 IPs (00.1% of total) 
1a:84:8d:a9:c4:17(ip-10-128-198-83.ec2.internal)     4105 IPs (00.4% of total) 
16:6b:0d:0f:fc:90(ip-10-128-222-185.ec2.internal)       64 IPs (00.0% of total) 
0a:55:01:96:29:9d(ip-10-128-235-0.ec2.internal)       10 IPs (00.0% of total) 
7e:1f:ef:1d:26:86(ip-10-128-236-11.ec2.internal)    14338 IPs (01.4% of total) 
56:de:40:12:bb:8d(ip-10-128-225-148.ec2.internal)    28215 IPs (02.7% of total) 
0a:f2:cb:be:ff:8e(ip-10-128-233-236.ec2.internal)       34 IPs (00.0% of total) 
8a:8b:d1:17:f7:33(ip-10-128-219-144.ec2.internal)    17399 IPs (01.7% of total) 
06:79:13:37:4e:9c(ip-10-128-204-95.ec2.internal)     8218 IPs (00.8% of total) 
96:7d:75:05:d9:db(ip-10-128-221-184.ec2.internal)     8199 IPs (00.8% of total) 
b2:e7:0e:fd:16:3c(ip-10-128-210-128.ec2.internal)       32 IPs (00.0% of total) 
ae:10:6f:36:96:ef(ip-10-128-231-9.ec2.internal)     4096 IPs (00.4% of total) 
72:a2:ef:9f:11:d6(ip-10-128-197-174.ec2.internal)       15 IPs (00.0% of total) 
7a:07:9a:90:f6:4d(ip-10-128-213-28.ec2.internal)     4385 IPs (00.4% of total) 
9a:5e:ca:97:75:1e(ip-10-128-197-65.ec2.internal)   179054 IPs (17.1% of total) 
1e:2b:8e:70:3e:f0(ip-10-128-216-176.ec2.internal)    30121 IPs (02.9% of total) 
7a:9c:fe:53:31:2c(ip-10-128-208-168.ec2.internal)    34764 IPs (03.3% of total) 
ae:8e:b3:e1:43:2f(ip-10-128-222-176.ec2.internal)    12664 IPs (01.2% of total) 
12:65:19:d1:38:db(ip-10-128-200-147.ec2.internal)     8203 IPs (00.8% of total) 
66:70:40:ae:e9:bd(ip-10-128-193-152.ec2.internal)       10 IPs (00.0% of total) 
3e:65:d5:e4:85:e8(ip-10-128-226-205.ec2.internal)    12554 IPs (01.2% of total) 
0a:9e:be:0d:d7:e4(ip-10-128-205-71.ec2.internal)    12288 IPs (01.2% of total) 
86:bc:eb:83:71:c4(ip-10-128-204-234.ec2.internal)     8061 IPs (00.8% of total) 
36:1f:70:99:66:63(ip-10-128-231-254.ec2.internal)    70821 IPs (06.8% of total) 
86:05:95:ef:89:5c(ip-10-128-235-185.ec2.internal)      768 IPs (00.1% of total) 
5a:57:4a:3a:e6:5d(ip-10-128-198-150.ec2.internal)    13325 IPs (01.3% of total) 
9e:f2:e0:3b:a3:fb(ip-10-128-221-77.ec2.internal)    12288 IPs (01.2% of total) 
ae:61:cd:c6:62:19(ip-10-128-211-129.ec2.internal)        8 IPs (00.0% of total) 
42:06:78:d0:55:9a(ip-10-128-208-59.ec2.internal)     8203 IPs (00.8% of total) 
36:75:46:6d:75:50(ip-10-128-200-85.ec2.internal)     8192 IPs (00.8% of total) 
92:2a:90:53:2a:04(ip-10-128-207-7.ec2.internal)      384 IPs (00.0% of total) 
42:9f:5b:98:56:2f(ip-10-128-230-233.ec2.internal)    24583 IPs (02.3% of total) 
56:1d:12:15:66:5b(ip-10-128-196-192.ec2.internal)      296 IPs (00.0% of total) 

/weave --local status connections

weave-net-pj8pg

-> 10.128.230.179:6783   established fastdp 82:29:dd:ef:df:75(ip-10-128-230-179.ec2.internal) mtu=1376
-> 10.128.213.28:6783    established fastdp 7a:07:9a:90:f6:4d(ip-10-128-213-28.ec2.internal) mtu=1376
<- 10.128.224.168:37896  established fastdp 9e:62:8c:9e:97:69(ip-10-128-224-168.ec2.internal) mtu=1376
-> 10.128.221.211:6783   established fastdp 66:d9:02:67:b0:3e(ip-10-128-221-211.ec2.internal) mtu=1376
-> 10.128.208.168:6783   established fastdp 7a:9c:fe:53:31:2c(ip-10-128-208-168.ec2.internal) mtu=1376
<- 10.128.225.148:56498  established fastdp 56:de:40:12:bb:8d(ip-10-128-225-148.ec2.internal) mtu=1376
-> 10.128.235.176:6783   established fastdp 32:bb:84:80:35:a4(ip-10-128-235-176.ec2.internal) mtu=1376
<- 10.128.200.90:33581   established fastdp d6:46:38:d6:2c:0e(ip-10-128-200-90.ec2.internal) mtu=1376
<- 10.128.239.45:41777   established fastdp e2:f3:fa:50:cd:67(ip-10-128-239-45.ec2.internal) mtu=1376
<- 10.128.192.116:39762  established fastdp 06:b3:f3:20:bc:be(ip-10-128-192-116.ec2.internal) mtu=1376
<- 10.128.197.65:43741   established fastdp 9a:5e:ca:97:75:1e(ip-10-128-197-65.ec2.internal) mtu=1376
<- 10.128.196.192:46219  established fastdp 56:1d:12:15:66:5b(ip-10-128-196-192.ec2.internal) mtu=1376
-> 10.128.208.59:6783    established fastdp 42:06:78:d0:55:9a(ip-10-128-208-59.ec2.internal) mtu=1376
-> 10.128.198.150:6783   established fastdp 5a:57:4a:3a:e6:5d(ip-10-128-198-150.ec2.internal) mtu=1376
<- 10.128.205.86:42675   established fastdp 9e:d3:58:e6:39:30(ip-10-128-205-86.ec2.internal) mtu=1376
-> 10.128.223.143:6783   established fastdp 6e:c8:93:2b:1b:8c(ip-10-128-223-143.ec2.internal) mtu=1376
-> 10.128.222.185:6783   established fastdp 16:6b:0d:0f:fc:90(ip-10-128-222-185.ec2.internal) mtu=1376
<- 10.128.207.7:49396    established fastdp 92:2a:90:53:2a:04(ip-10-128-207-7.ec2.internal) mtu=1376
-> 10.128.222.176:6783   established fastdp ae:8e:b3:e1:43:2f(ip-10-128-222-176.ec2.internal) mtu=1376
-> 10.128.230.233:6783   established fastdp 42:9f:5b:98:56:2f(ip-10-128-230-233.ec2.internal) mtu=1376
<- 10.128.209.244:50361  established fastdp 6a:f0:40:bb:b4:ce(ip-10-128-209-244.ec2.internal) mtu=1376
<- 10.128.216.176:51166  established fastdp 1e:2b:8e:70:3e:f0(ip-10-128-216-176.ec2.internal) mtu=1376
-> 10.128.196.11:6783    established fastdp a6:ec:c3:0a:97:ef(ip-10-128-196-11.ec2.internal) mtu=1376
<- 10.128.204.234:34518  established fastdp 86:bc:eb:83:71:c4(ip-10-128-204-234.ec2.internal) mtu=1376
-> 10.128.219.144:6783   established fastdp 8a:8b:d1:17:f7:33(ip-10-128-219-144.ec2.internal) mtu=1376
<- 10.128.223.78:59972   established fastdp de:de:81:69:3b:68(ip-10-128-223-78.ec2.internal) mtu=1376
<- 10.128.225.55:35791   established fastdp 26:f5:1a:4e:36:68(ip-10-128-225-55.ec2.internal) mtu=1376
<- 10.128.231.9:51056    established fastdp ae:10:6f:36:96:ef(ip-10-128-231-9.ec2.internal) mtu=1376
-> 10.128.211.129:6783   established fastdp ae:61:cd:c6:62:19(ip-10-128-211-129.ec2.internal) mtu=1376
<- 10.128.208.254:34195  established fastdp 1a:6e:d5:21:cb:5d(ip-10-128-208-254.ec2.internal) mtu=1376
<- 10.128.201.80:43451   established fastdp 4a:83:c5:7a:e0:53(ip-10-128-201-80.ec2.internal) mtu=1376
-> 10.128.217.187:6783   established fastdp e2:3d:21:ef:02:5f(ip-10-128-217-187.ec2.internal) mtu=1376
-> 10.128.234.112:6783   established fastdp de:f8:54:f2:0c:28(ip-10-128-234-112.ec2.internal) mtu=1376
-> 10.128.200.147:6783   established fastdp 12:65:19:d1:38:db(ip-10-128-200-147.ec2.internal) mtu=1376
-> 10.128.235.185:6783   established fastdp 86:05:95:ef:89:5c(ip-10-128-235-185.ec2.internal) mtu=1376
-> 10.128.234.57:6783    established fastdp 0e:d4:f3:53:1c:91(ip-10-128-234-57.ec2.internal) mtu=1376
-> 10.128.200.85:6783    established fastdp 36:75:46:6d:75:50(ip-10-128-200-85.ec2.internal) mtu=1376
-> 10.128.226.205:6783   established fastdp 3e:65:d5:e4:85:e8(ip-10-128-226-205.ec2.internal) mtu=1376
-> 10.128.221.77:6783    established fastdp 9e:f2:e0:3b:a3:fb(ip-10-128-221-77.ec2.internal) mtu=1376
<- 10.128.235.0:34197    established fastdp 0a:55:01:96:29:9d(ip-10-128-235-0.ec2.internal) mtu=1376
-> 10.128.198.83:6783    established fastdp 1a:84:8d:a9:c4:17(ip-10-128-198-83.ec2.internal) mtu=1376
-> 10.128.205.71:6783    established fastdp 0a:9e:be:0d:d7:e4(ip-10-128-205-71.ec2.internal) mtu=1376
-> 10.128.204.95:6783    established fastdp 06:79:13:37:4e:9c(ip-10-128-204-95.ec2.internal) mtu=1376
<- 10.128.206.176:54665  established fastdp 5e:58:a6:d8:43:31(ip-10-128-206-176.ec2.internal) mtu=1376
-> 10.128.197.174:6783   established fastdp 72:a2:ef:9f:11:d6(ip-10-128-197-174.ec2.internal) mtu=1376
-> 10.128.233.236:6783   established fastdp 0a:f2:cb:be:ff:8e(ip-10-128-233-236.ec2.internal) mtu=1376
-> 10.128.221.184:6783   established fastdp 96:7d:75:05:d9:db(ip-10-128-221-184.ec2.internal) mtu=1376
-> 10.128.236.11:6783    established fastdp 7e:1f:ef:1d:26:86(ip-10-128-236-11.ec2.internal) mtu=1376
<- 10.128.193.152:47586  established fastdp 66:70:40:ae:e9:bd(ip-10-128-193-152.ec2.internal) mtu=1376
-> 10.128.214.51:6783    established fastdp 42:73:b4:55:6e:9d(ip-10-128-214-51.ec2.internal) mtu=1376
-> 10.128.210.128:6783   established fastdp b2:e7:0e:fd:16:3c(ip-10-128-210-128.ec2.internal) mtu=1376
-> 10.128.193.210:6783   established fastdp be:38:56:4d:00:02(ip-10-128-193-210.ec2.internal) mtu=1376
<- 10.128.231.254:47372  established fastdp 36:1f:70:99:66:63(ip-10-128-231-254.ec2.internal) mtu=1376
-> 10.128.203.252:6783   failed      cannot connect to ourself, retry: never

weave-net-24nkq

<- 10.128.231.254:53321  established fastdp 36:1f:70:99:66:63(ip-10-128-231-254.ec2.internal) mtu=1376
-> 10.128.223.143:6783   established fastdp 6e:c8:93:2b:1b:8c(ip-10-128-223-143.ec2.internal) mtu=1376
-> 10.128.198.83:6783    established fastdp 1a:84:8d:a9:c4:17(ip-10-128-198-83.ec2.internal) mtu=1376
<- 10.128.230.233:34352  established fastdp 42:9f:5b:98:56:2f(ip-10-128-230-233.ec2.internal) mtu=1376
<- 10.128.235.176:43004  established fastdp 32:bb:84:80:35:a4(ip-10-128-235-176.ec2.internal) mtu=1376
-> 10.128.221.184:6783   established fastdp 96:7d:75:05:d9:db(ip-10-128-221-184.ec2.internal) mtu=1376
-> 10.128.230.179:6783   established fastdp 82:29:dd:ef:df:75(ip-10-128-230-179.ec2.internal) mtu=1376
<- 10.128.197.65:36222   established fastdp 9a:5e:ca:97:75:1e(ip-10-128-197-65.ec2.internal) mtu=1376
-> 10.128.211.129:6783   established fastdp ae:61:cd:c6:62:19(ip-10-128-211-129.ec2.internal) mtu=1376
<- 10.128.208.254:45153  established fastdp 1a:6e:d5:21:cb:5d(ip-10-128-208-254.ec2.internal) mtu=1376
<- 10.128.219.144:39217  established fastdp 8a:8b:d1:17:f7:33(ip-10-128-219-144.ec2.internal) mtu=1376
<- 10.128.239.45:44095   established fastdp e2:f3:fa:50:cd:67(ip-10-128-239-45.ec2.internal) mtu=1376
-> 10.128.233.236:6783   established fastdp 0a:f2:cb:be:ff:8e(ip-10-128-233-236.ec2.internal) mtu=1376
-> 10.128.193.152:6783   established fastdp 66:70:40:ae:e9:bd(ip-10-128-193-152.ec2.internal) mtu=1376
-> 10.128.192.116:6783   established fastdp 06:b3:f3:20:bc:be(ip-10-128-192-116.ec2.internal) mtu=1376
<- 10.128.208.59:41583   established fastdp 42:06:78:d0:55:9a(ip-10-128-208-59.ec2.internal) mtu=1376
-> 10.128.217.187:6783   established fastdp e2:3d:21:ef:02:5f(ip-10-128-217-187.ec2.internal) mtu=1376
-> 10.128.204.95:6783    established fastdp 06:79:13:37:4e:9c(ip-10-128-204-95.ec2.internal) mtu=1376
-> 10.128.234.112:6783   established fastdp de:f8:54:f2:0c:28(ip-10-128-234-112.ec2.internal) mtu=1376
<- 10.128.207.7:35935    established fastdp 92:2a:90:53:2a:04(ip-10-128-207-7.ec2.internal) mtu=1376
<- 10.128.200.85:47042   established fastdp 36:75:46:6d:75:50(ip-10-128-200-85.ec2.internal) mtu=1376
<- 10.128.193.210:50198  established fastdp be:38:56:4d:00:02(ip-10-128-193-210.ec2.internal) mtu=1376
<- 10.128.196.192:51459  established fastdp 56:1d:12:15:66:5b(ip-10-128-196-192.ec2.internal) mtu=1376
-> 10.128.225.55:6783    established fastdp 26:f5:1a:4e:36:68(ip-10-128-225-55.ec2.internal) mtu=1376
-> 10.128.235.0:6783     established fastdp 0a:55:01:96:29:9d(ip-10-128-235-0.ec2.internal) mtu=1376
<- 10.128.205.71:34427   established fastdp 0a:9e:be:0d:d7:e4(ip-10-128-205-71.ec2.internal) mtu=1376
-> 10.128.208.168:6783   established fastdp 7a:9c:fe:53:31:2c(ip-10-128-208-168.ec2.internal) mtu=1376
<- 10.128.222.176:42527  established fastdp ae:8e:b3:e1:43:2f(ip-10-128-222-176.ec2.internal) mtu=1376
<- 10.128.200.90:44783   established fastdp d6:46:38:d6:2c:0e(ip-10-128-200-90.ec2.internal) mtu=1376
-> 10.128.226.205:6783   established fastdp 3e:65:d5:e4:85:e8(ip-10-128-226-205.ec2.internal) mtu=1376
-> 10.128.221.211:6783   established fastdp 66:d9:02:67:b0:3e(ip-10-128-221-211.ec2.internal) mtu=1376
-> 10.128.206.176:6783   established fastdp 5e:58:a6:d8:43:31(ip-10-128-206-176.ec2.internal) mtu=1376
<- 10.128.196.11:54126   established fastdp a6:ec:c3:0a:97:ef(ip-10-128-196-11.ec2.internal) mtu=1376
-> 10.128.213.28:6783    established fastdp 7a:07:9a:90:f6:4d(ip-10-128-213-28.ec2.internal) mtu=1376
<- 10.128.221.77:38395   established fastdp 9e:f2:e0:3b:a3:fb(ip-10-128-221-77.ec2.internal) mtu=1376
<- 10.128.235.185:48855  established fastdp 86:05:95:ef:89:5c(ip-10-128-235-185.ec2.internal) mtu=1376
<- 10.128.222.185:49797  established fastdp 16:6b:0d:0f:fc:90(ip-10-128-222-185.ec2.internal) mtu=1376
<- 10.128.234.57:50655   established fastdp 0e:d4:f3:53:1c:91(ip-10-128-234-57.ec2.internal) mtu=1376
-> 10.128.200.147:6783   established fastdp 12:65:19:d1:38:db(ip-10-128-200-147.ec2.internal) mtu=1376
-> 10.128.214.51:6783    established fastdp 42:73:b4:55:6e:9d(ip-10-128-214-51.ec2.internal) mtu=1376
-> 10.128.203.252:6783   established fastdp b2:f8:2e:00:01:60(ip-10-128-203-252.ec2.internal) mtu=1376
<- 10.128.236.11:41005   established fastdp 7e:1f:ef:1d:26:86(ip-10-128-236-11.ec2.internal) mtu=1376
-> 10.128.209.244:6783   established fastdp 6a:f0:40:bb:b4:ce(ip-10-128-209-244.ec2.internal) mtu=1376
-> 10.128.197.174:6783   established fastdp 72:a2:ef:9f:11:d6(ip-10-128-197-174.ec2.internal) mtu=1376
<- 10.128.224.168:33084  established fastdp 9e:62:8c:9e:97:69(ip-10-128-224-168.ec2.internal) mtu=1376
<- 10.128.204.234:53347  established fastdp 86:bc:eb:83:71:c4(ip-10-128-204-234.ec2.internal) mtu=1376
-> 10.128.198.150:6783   established fastdp 5a:57:4a:3a:e6:5d(ip-10-128-198-150.ec2.internal) mtu=1376
-> 10.128.216.176:6783   established fastdp 1e:2b:8e:70:3e:f0(ip-10-128-216-176.ec2.internal) mtu=1376
<- 10.128.231.9:34922    established fastdp ae:10:6f:36:96:ef(ip-10-128-231-9.ec2.internal) mtu=1376
-> 10.128.205.86:6783    established fastdp 9e:d3:58:e6:39:30(ip-10-128-205-86.ec2.internal) mtu=1376
<- 10.128.210.128:41242  established fastdp b2:e7:0e:fd:16:3c(ip-10-128-210-128.ec2.internal) mtu=1376
-> 10.128.201.80:6783    established fastdp 4a:83:c5:7a:e0:53(ip-10-128-201-80.ec2.internal) mtu=1376
-> 10.128.225.148:6783   established fastdp 56:de:40:12:bb:8d(ip-10-128-225-148.ec2.internal) mtu=1376
-> 10.128.223.78:6783    failed      cannot connect to ourself, retry: never

weave-net-hd9dv

<- 10.128.193.152:49632  established fastdp 66:70:40:ae:e9:bd(ip-10-128-193-152.ec2.internal) mtu=1376
<- 10.128.200.85:43491   established fastdp 36:75:46:6d:75:50(ip-10-128-200-85.ec2.internal) mtu=1376
<- 10.128.225.148:32812  established fastdp 56:de:40:12:bb:8d(ip-10-128-225-148.ec2.internal) mtu=1376
<- 10.128.239.45:50165   established fastdp e2:f3:fa:50:cd:67(ip-10-128-239-45.ec2.internal) mtu=1376
-> 10.128.223.78:6783    established fastdp de:de:81:69:3b:68(ip-10-128-223-78.ec2.internal) mtu=1376
<- 10.128.197.65:47127   established fastdp 9a:5e:ca:97:75:1e(ip-10-128-197-65.ec2.internal) mtu=1376
<- 10.128.222.176:47534  established fastdp ae:8e:b3:e1:43:2f(ip-10-128-222-176.ec2.internal) mtu=1376
<- 10.128.233.236:32899  established fastdp 0a:f2:cb:be:ff:8e(ip-10-128-233-236.ec2.internal) mtu=1376
<- 10.128.221.77:48835   established fastdp 9e:f2:e0:3b:a3:fb(ip-10-128-221-77.ec2.internal) mtu=1376
<- 10.128.225.55:35895   established fastdp 26:f5:1a:4e:36:68(ip-10-128-225-55.ec2.internal) mtu=1376
-> 10.128.231.254:6783   established fastdp 36:1f:70:99:66:63(ip-10-128-231-254.ec2.internal) mtu=1376
<- 10.128.223.143:49458  established fastdp 6e:c8:93:2b:1b:8c(ip-10-128-223-143.ec2.internal) mtu=1376
-> 10.128.198.150:6783   established fastdp 5a:57:4a:3a:e6:5d(ip-10-128-198-150.ec2.internal) mtu=1376
-> 10.128.208.254:6783   established fastdp 1a:6e:d5:21:cb:5d(ip-10-128-208-254.ec2.internal) mtu=1376
-> 10.128.214.51:6783    established fastdp 42:73:b4:55:6e:9d(ip-10-128-214-51.ec2.internal) mtu=1376
-> 10.128.200.147:6783   established fastdp 12:65:19:d1:38:db(ip-10-128-200-147.ec2.internal) mtu=1376
<- 10.128.200.90:48080   established fastdp d6:46:38:d6:2c:0e(ip-10-128-200-90.ec2.internal) mtu=1376
-> 10.128.196.11:6783    established fastdp a6:ec:c3:0a:97:ef(ip-10-128-196-11.ec2.internal) mtu=1376
<- 10.128.224.168:55828  established fastdp 9e:62:8c:9e:97:69(ip-10-128-224-168.ec2.internal) mtu=1376
-> 10.128.235.176:6783   established fastdp 32:bb:84:80:35:a4(ip-10-128-235-176.ec2.internal) mtu=1376
<- 10.128.201.80:43522   established fastdp 4a:83:c5:7a:e0:53(ip-10-128-201-80.ec2.internal) mtu=1376
-> 10.128.211.129:6783   established fastdp ae:61:cd:c6:62:19(ip-10-128-211-129.ec2.internal) mtu=1376
-> 10.128.209.244:6783   established fastdp 6a:f0:40:bb:b4:ce(ip-10-128-209-244.ec2.internal) mtu=1376
<- 10.128.235.0:52484    established fastdp 0a:55:01:96:29:9d(ip-10-128-235-0.ec2.internal) mtu=1376
-> 10.128.207.7:6783     established fastdp 92:2a:90:53:2a:04(ip-10-128-207-7.ec2.internal) mtu=1376
<- 10.128.197.174:58409  established fastdp 72:a2:ef:9f:11:d6(ip-10-128-197-174.ec2.internal) mtu=1376
-> 10.128.204.95:6783    established fastdp 06:79:13:37:4e:9c(ip-10-128-204-95.ec2.internal) mtu=1376
<- 10.128.221.211:55497  established fastdp 66:d9:02:67:b0:3e(ip-10-128-221-211.ec2.internal) mtu=1376
<- 10.128.206.176:56035  established fastdp 5e:58:a6:d8:43:31(ip-10-128-206-176.ec2.internal) mtu=1376
-> 10.128.208.59:6783    established fastdp 42:06:78:d0:55:9a(ip-10-128-208-59.ec2.internal) mtu=1376
<- 10.128.192.116:39428  established fastdp 06:b3:f3:20:bc:be(ip-10-128-192-116.ec2.internal) mtu=1376
-> 10.128.216.176:6783   established fastdp 1e:2b:8e:70:3e:f0(ip-10-128-216-176.ec2.internal) mtu=1376
-> 10.128.221.184:6783   established fastdp 96:7d:75:05:d9:db(ip-10-128-221-184.ec2.internal) mtu=1376
<- 10.128.235.185:49155  established fastdp 86:05:95:ef:89:5c(ip-10-128-235-185.ec2.internal) mtu=1376
-> 10.128.205.86:6783    established fastdp 9e:d3:58:e6:39:30(ip-10-128-205-86.ec2.internal) mtu=1376
<- 10.128.226.205:41274  established fastdp 3e:65:d5:e4:85:e8(ip-10-128-226-205.ec2.internal) mtu=1376
<- 10.128.203.252:37893  established fastdp b2:f8:2e:00:01:60(ip-10-128-203-252.ec2.internal) mtu=1376
-> 10.128.208.168:6783   established fastdp 7a:9c:fe:53:31:2c(ip-10-128-208-168.ec2.internal) mtu=1376
-> 10.128.205.71:6783    established fastdp 0a:9e:be:0d:d7:e4(ip-10-128-205-71.ec2.internal) mtu=1376
<- 10.128.222.185:57815  established fastdp 16:6b:0d:0f:fc:90(ip-10-128-222-185.ec2.internal) mtu=1376
-> 10.128.198.83:6783    established fastdp 1a:84:8d:a9:c4:17(ip-10-128-198-83.ec2.internal) mtu=1376
-> 10.128.196.192:6783   established fastdp 56:1d:12:15:66:5b(ip-10-128-196-192.ec2.internal) mtu=1376
-> 10.128.204.234:6783   established fastdp 86:bc:eb:83:71:c4(ip-10-128-204-234.ec2.internal) mtu=1376
-> 10.128.217.187:6783   established fastdp e2:3d:21:ef:02:5f(ip-10-128-217-187.ec2.internal) mtu=1376
<- 10.128.234.112:46161  established fastdp de:f8:54:f2:0c:28(ip-10-128-234-112.ec2.internal) mtu=1376
-> 10.128.213.28:6783    established fastdp 7a:07:9a:90:f6:4d(ip-10-128-213-28.ec2.internal) mtu=1376
<- 10.128.236.11:44038   established fastdp 7e:1f:ef:1d:26:86(ip-10-128-236-11.ec2.internal) mtu=1376
<- 10.128.210.128:59630  established fastdp b2:e7:0e:fd:16:3c(ip-10-128-210-128.ec2.internal) mtu=1376
<- 10.128.230.233:43139  established fastdp 42:9f:5b:98:56:2f(ip-10-128-230-233.ec2.internal) mtu=1376
<- 10.128.219.144:60392  established fastdp 8a:8b:d1:17:f7:33(ip-10-128-219-144.ec2.internal) mtu=1376
-> 10.128.193.210:6783   established fastdp be:38:56:4d:00:02(ip-10-128-193-210.ec2.internal) mtu=1376
-> 10.128.230.179:6783   established fastdp 82:29:dd:ef:df:75(ip-10-128-230-179.ec2.internal) mtu=1376
-> 10.128.231.9:6783     established fastdp ae:10:6f:36:96:ef(ip-10-128-231-9.ec2.internal) mtu=1376
-> 10.128.234.57:6783    failed      cannot connect to ourself, retry: never

logs from
weave-net-pj8pg
weave-net-24nkq
weave-net-hd9dv
I've tried to recreate masters one by one (it's asg-based kops three master installation) but those messages do not go away.

What's the right way of resolving this issue?
@bboreham
Copy link
Contributor

@kostyrev the latest code (not in a release version) should recover from this much better, but in the meantime take a look at #3310 (comment)

Also consider if you have anything that might be abruply killing the Weave Net containers - a memory limit or liveness probe. This could be the trigger for the problem.

@kostyrev
Copy link
Author

kostyrev commented Oct 17, 2019 •
edited
Loading

thanks!
but that comment suggests removing db in /var/lib/weave but isn't killing master one by one suppose to have the same effect? I did it and it didn't resolve the problem.

these are new recreated masters and weave-net pods were not restarted

> kgpo -l name=weave-net -owide | egrep '(ip-10-128-203-252|ip-10-128-223-78|ip-10-128-234-57)'
weave-net-24nkq   2/2     Running   0          133m    10.128.223.78    ip-10-128-223-78.ec2.internal    <none>           <none>
weave-net-hd9dv   2/2     Running   0          116m    10.128.234.57    ip-10-128-234-57.ec2.internal    <none>           <none>
weave-net-pj8pg   2/2     Running   0          124m    10.128.203.252   ip-10-128-203-252.ec2.internal   <none>           <none>

@bboreham
Copy link
Contributor

You mean you removed and recreated the entire machine? Yes, that should make the file go away.

I guess it's possible that each time you recreated one node, you left two nodes running with an inconsistent state, and each time the new node came up it would accept one of those states, so you maintain the inconsistency across all three restarts. Something like the story of the fox, the chicken and the corn.

Hopefully there is an ordering where you can remove the state, restart, and lose the inconsistency.

@kostyrev
Copy link
Author

kostyrev commented Oct 17, 2019 •
edited
Loading

You mean you removed and recreated the entire machine?

yes

I'll try to recreate them all at the same time tomorrow.

thank you!

@kostyrev
Copy link
Author

so I found verify-weave.sh from this issue and did how issue suggests.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kostyrev @bboreham