This repository has been archived by the owner on Jun 20, 2024. It is now read-only.
Nodes are attempting to claim same IP range #3310
Comments
|
@zacblazic Thanks for the issue. Regarding the first issue, do you have by any chance full Weave Net logs from The second issue is going to be fixed in #3317. |
|
@brb I'll try get them for you, no guarantees though. We're seeing this happen more and more often now, seemingly random but cannot confirm the catalyst at this time. Regarding the second issue, is there a way to manually remove the annotation or clear this state? It's clearly something that persists across the weave cluster as fresh nodes are having the same problem. |
|
@brb I work with Zac, and our cluster is getting unstable at least once a day. We fix it by finding out the bad weave pod and clearing it's state. This only started happening after upgrading to 2.3.0. We never had this issue with 2.2.0. This is the result of connection status on all weave pods 👇 ... you can see it's pretty bad (although better than what we've seen recently). |
|
@itskingori thanks; we really would like logs (as in If you have cleared a pod's state and restarted it and then it goes bad within the day then there is a good chance the logs do have the vital information. If the pod has restarted please fetch the previous logs too with To be clear, in case of message |
|
@bboreham Got it! Will keep you guys posted. |
|
@bboreham @brb While it hasn't happened again, I've done some investigation and I'd like to pass a theory past you two. Could it be as a result of the pods being abruptly killed .e.g. an OOM? In the case of #3310 (comment), we cleared the state of You can see that
We installed via kops, therefore our resource requests and limits are set to 👇 resources:
limits:
memory: 200Mi
requests:
cpu: 50m
memory: 200MiHere are the logs of
And here are the logs of
You can see the logging profile of
And the logging profile of
So:
|
|
@bboreham @brb Another theory, could it be that the infinite loop fixed in #3317 is causing the weave pod to use more and more memory? See 30 days usage below:
My colleague @tsu-shiuan, thinks that increasing memory would be hiding the problem ... and that while the OOMs are likely the cause of the issue described in #3310 (comment), that there's an underlying problem causing the increase in memory. Some points to note:
The disruption right after the 28th is us rotating nodes, so that's expected ... but you can see after that things are not looking good. |
|
@zacblazic @itskingori Thanks for the info.
I doubt that any change from 2.2.0 to 2.3.0 could have triggered the problem (maybe "Build Weave Net with Go 1.10.1 #3273", but let's keep it as a last resort). My guess is that the upgrade restarted Weave Net on each node which triggered the faulty reclaim
I don't have a cmd at hand, but try updating the annotation with Anyway, do you have any logs for
Might be that boltdb which stores IPAM fails to write to disk as the process gets killed. So, after the restart Weave Net restores a stale data (@bboreham shouldn't the CRDT IPAM ring be able to merge in such situation anyway?). Interested in your logs after you have reproduced the issue. Just please upload them as txt to make grepping possible. Ta. |
|
I can see exactly same kind of error |
|
@kuznero please open your own issue and supply logs. |
|
OK, I didn't read the whole question, sorry.
Something like this sequence?
To avoid this, we persist to disk before sending the message from I guess it's possible there is a bug in the above. Or it's something else. |
|
@zacblazic @itskingori @kuznero What file system (with options) is of the @kuznero Can you easily reproduce the issue? |
@bboreham @brb I think I might have the logs of a pod that's just started and is exhibiting this behaviour. You can see that it started slightly past 23:25 👇
And started getting the "Received update for IP range I own" error a little after 👇
These are the full logs 👉 weave-net-z5lmp.log Yet it's connections look ok 👇 $ kubectl exec -it weave-net-z5lmp -n kube-system /bin/sh
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-z5lmp -n kube-system' to see all of the containers in this pod.
/home/weave # ./weave --local status connections
-> 10.80.102.195:6783 established fastdp 56:4c:56:6f:f5:90(ip-10-80-102-195.ec2.internal) mtu=1376
<- 10.80.67.160:32281 established fastdp 3e:8f:f3:15:0c:e6(ip-10-80-67-160.ec2.internal) mtu=1376
-> 10.80.45.255:6783 established fastdp 6a:7d:84:55:63:40(ip-10-80-45-255.ec2.internal) mtu=1376
-> 10.80.110.250:6783 established fastdp 02:8f:3d:02:22:b2(ip-10-80-110-250.ec2.internal) mtu=1376
-> 10.80.91.69:6783 established fastdp a2:89:81:f0:f7:cb(ip-10-80-91-69.ec2.internal) mtu=1376
-> 10.80.112.109:6783 established fastdp 72:d3:22:05:8f:08(ip-10-80-112-109.ec2.internal) mtu=1376
-> 10.80.66.231:6783 established fastdp 52:ff:e3:6c:74:1d(ip-10-80-66-231.ec2.internal) mtu=1376
<- 10.80.112.120:28928 established fastdp be:10:d6:f0:eb:68(ip-10-80-112-120.ec2.internal) mtu=1376
-> 10.80.90.166:6783 established fastdp d2:fe:ff:e6:ca:a7(ip-10-80-90-166.ec2.internal) mtu=1376
<- 10.80.61.78:40629 established fastdp c2:05:23:81:b9:b1(ip-10-80-61-78.ec2.internal) mtu=1376
<- 10.80.44.252:13551 established fastdp 32:c1:c8:45:6e:0d(ip-10-80-44-252.ec2.internal) mtu=1376
-> 10.80.53.166:6783 established fastdp 9a:6b:6f:33:1c:60(ip-10-80-53-166.ec2.internal) mtu=1376
<- 10.80.89.245:49733 established fastdp 0a:e3:03:b2:57:8c(ip-10-80-89-245.ec2.internal) mtu=1376
-> 10.80.59.37:6783 established fastdp 9a:49:8b:6b:a8:12(ip-10-80-59-37.ec2.internal) mtu=1376
-> 10.80.67.172:6783 established fastdp 4a:f9:d0:15:61:a7(ip-10-80-67-172.ec2.internal) mtu=1376
-> 10.80.65.216:6783 established fastdp 42:90:28:9a:2c:a2(ip-10-80-65-216.ec2.internal) mtu=1376
<- 10.80.114.8:52196 established fastdp 6e:0e:51:4f:66:9e(ip-10-80-114-8.ec2.internal) mtu=1376
<- 10.80.96.210:25680 established fastdp 96:36:1a:d1:59:07(ip-10-80-96-210.ec2.internal) mtu=1376
-> 10.80.41.90:6783 established fastdp a2:63:98:df:ee:d0(ip-10-80-41-90.ec2.internal) mtu=1376
-> 10.80.120.250:6783 established fastdp e6:19:2c:86:42:27(ip-10-80-120-250.ec2.internal) mtu=1376
-> 10.80.125.5:6783 established fastdp 8e:b1:f9:dc:6b:31(ip-10-80-125-5.ec2.internal) mtu=1376
-> 10.80.54.76:6783 established fastdp b2:ef:a5:21:f9:a3(ip-10-80-54-76.ec2.internal) mtu=1376
-> 10.80.37.248:6783 failed cannot connect to ourself, retry: neverAnd ipam status doesn't look too bad 👇 $ kubectl exec -it weave-net-z5lmp -n kube-system /bin/sh
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-z5lmp -n kube-system' to see all of the containers in this pod.
/home/weave # ./weave --local status ipam
ca:b9:26:99:63:b4(ip-10-80-37-248.ec2.internal) 135880 IPs (06.5% of total) (4 active)
d6:c3:9a:8e:74:f0(ip-10-80-105-52.ec2.internal) 65536 IPs (03.1% of total) - unreachable!
8a:23:63:a2:3b:04(ip-10-80-112-98.ec2.internal) 32768 IPs (01.6% of total) - unreachable!
0a:ed:0e:50:ed:f9(ip-10-80-108-104.ec2.internal) 65536 IPs (03.1% of total) - unreachable!
6a:7d:84:55:63:40(ip-10-80-45-255.ec2.internal) 108236 IPs (05.2% of total)
a2:89:81:f0:f7:cb(ip-10-80-91-69.ec2.internal) 91695 IPs (04.4% of total)
9a:6b:6f:33:1c:60(ip-10-80-53-166.ec2.internal) 47608 IPs (02.3% of total)
a2:63:98:df:ee:d0(ip-10-80-41-90.ec2.internal) 44136 IPs (02.1% of total)
e2:09:61:37:d0:9e(ip-10-80-46-186.ec2.internal) 65536 IPs (03.1% of total) - unreachable!
5e:80:0c:4b:9a:41(ip-10-80-120-27.ec2.internal) 8192 IPs (00.4% of total) - unreachable!
c2:05:23:81:b9:b1(ip-10-80-61-78.ec2.internal) 83 IPs (00.0% of total)
5a:47:78:0d:74:53(ip-10-80-104-144.ec2.internal) 32768 IPs (01.6% of total) - unreachable!
56:4c:56:6f:f5:90(ip-10-80-102-195.ec2.internal) 88952 IPs (04.2% of total)
52:ff:e3:6c:74:1d(ip-10-80-66-231.ec2.internal) 106296 IPs (05.1% of total)
e6:19:2c:86:42:27(ip-10-80-120-250.ec2.internal) 32768 IPs (01.6% of total)
8e:b1:f9:dc:6b:31(ip-10-80-125-5.ec2.internal) 8192 IPs (00.4% of total)
ba:5f:5d:8e:c5:49(ip-10-80-102-17.ec2.internal) 16384 IPs (00.8% of total) - unreachable!
62:49:f3:21:c1:b3(ip-10-80-57-2.ec2.internal) 32768 IPs (01.6% of total) - unreachable!
7a:26:75:22:4c:a3(ip-10-80-55-0.ec2.internal) 65536 IPs (03.1% of total) - unreachable!
02:f2:c2:33:21:a8(ip-10-80-89-194.ec2.internal) 32768 IPs (01.6% of total) - unreachable!
72:d3:22:05:8f:08(ip-10-80-112-109.ec2.internal) 65536 IPs (03.1% of total)
62:4b:82:bc:ee:4a(ip-10-80-106-92.ec2.internal) 32768 IPs (01.6% of total) - unreachable!
c2:08:11:3c:6b:02(ip-10-80-44-87.ec2.internal) 65536 IPs (03.1% of total) - unreachable!
3e:8f:f3:15:0c:e6(ip-10-80-67-160.ec2.internal) 24 IPs (00.0% of total)
16:f0:47:37:8a:18(ip-10-80-101-176.ec2.internal) 65536 IPs (03.1% of total) - unreachable!
fa:d3:e4:a1:47:54(ip-10-80-74-138.ec2.internal) 32768 IPs (01.6% of total) - unreachable!
0a:e3:03:b2:57:8c(ip-10-80-89-245.ec2.internal) 56116 IPs (02.7% of total)
be:10:d6:f0:eb:68(ip-10-80-112-120.ec2.internal) 89798 IPs (04.3% of total)
6e:0e:51:4f:66:9e(ip-10-80-114-8.ec2.internal) 115398 IPs (05.5% of total)
9a:49:8b:6b:a8:12(ip-10-80-59-37.ec2.internal) 98216 IPs (04.7% of total)
4a:f9:d0:15:61:a7(ip-10-80-67-172.ec2.internal) 8304 IPs (00.4% of total)
42:90:28:9a:2c:a2(ip-10-80-65-216.ec2.internal) 52000 IPs (02.5% of total)
32:c1:c8:45:6e:0d(ip-10-80-44-252.ec2.internal) 16384 IPs (00.8% of total)
ea:0c:50:90:93:c3(ip-10-80-46-212.ec2.internal) 16384 IPs (00.8% of total) - unreachable!
9e:ce:d8:bd:e6:8d(ip-10-80-124-112.ec2.internal) 32768 IPs (01.6% of total) - unreachable!
96:36:1a:d1:59:07(ip-10-80-96-210.ec2.internal) 16384 IPs (00.8% of total)
b2:ef:a5:21:f9:a3(ip-10-80-54-76.ec2.internal) 87747 IPs (04.2% of total)
02:8f:3d:02:22:b2(ip-10-80-110-250.ec2.internal) 65543 IPs (03.1% of total)
fe:0f:36:cd:88:d8(ip-10-80-40-99.ec2.internal) 16384 IPs (00.8% of total) - unreachable!
1a:d9:db:f0:74:a4(ip-10-80-63-127.ec2.internal) 65536 IPs (03.1% of total) - unreachable!
26:be:94:88:07:51(ip-10-80-63-133.ec2.internal) 16384 IPs (00.8% of total) - unreachable!
Host filesystem 👇 $ mount | grep /dev/xvda1
/dev/xvda1 on / type ext4 (rw,relatime,data=ordered)
/dev/xvda1 on /var/lib/docker/overlay2 type ext4 (rw,relatime,data=ordered)
/dev/xvda1 on /var/lib/kubelet/pods/a0ea4d18-ab00-11e8-ac1a-06f70fba1e78/volume-subpaths/telegraf-configuration/telegraf/0 type ext4 (ro,relatime,data=ordered)
/dev/xvda1 on /var/lib/kubelet/pods/a0e24f65-ab00-11e8-ac1a-06f70fba1e78/volume-subpaths/filebeat-configuration/filebeat/2 type ext4 (ro,relatime,data=ordered) |
@brb Sorry that I'm so late on this, but I've got some interesting findings. $ kubectl get configmap -n=kube-system -oyaml weave-net
...
{
"Peers": [
{
"PeerName": "da:3e:d4:a5:e9:42",
"NodeName": "ip-10-80-78-181.ec2.internal"
},
{
"PeerName": "0a:e3:03:b2:57:8c",
"NodeName": "ip-10-80-89-245.ec2.internal"
},
{
"PeerName": "9a:49:8b:6b:a8:12",
"NodeName": "ip-10-80-59-37.ec2.internal"
},
{
"PeerName": "56:4c:56:6f:f5:90",
"NodeName": "ip-10-80-102-195.ec2.internal"
},
{
"PeerName": "9a:6b:6f:33:1c:60",
"NodeName": "ip-10-80-53-166.ec2.internal"
},
{
"PeerName": "a2:89:81:f0:f7:cb",
"NodeName": "ip-10-80-91-69.ec2.internal"
},
{
"PeerName": "be:10:d6:f0:eb:68",
"NodeName": "ip-10-80-112-120.ec2.internal"
},
{
"PeerName": "a2:63:98:df:ee:d0",
"NodeName": "ip-10-80-41-90.ec2.internal"
},
{
"PeerName": "6e:0e:51:4f:66:9e",
"NodeName": "ip-10-80-114-8.ec2.internal"
},
{
"PeerName": "6a:7d:84:55:63:40",
"NodeName": "ip-10-80-45-255.ec2.internal"
},
{
"PeerName": "3e:8f:f3:15:0c:e6",
"NodeName": "ip-10-80-67-160.ec2.internal"
},
{
"PeerName": "b2:ef:a5:21:f9:a3",
"NodeName": "ip-10-80-54-76.ec2.internal"
},
{
"PeerName": "52:ff:e3:6c:74:1d",
"NodeName": "ip-10-80-66-231.ec2.internal"
},
{
"PeerName": "42:90:28:9a:2c:a2",
"NodeName": "ip-10-80-65-216.ec2.internal"
},
{
"PeerName": "4a:f9:d0:15:61:a7",
"NodeName": "ip-10-80-67-172.ec2.internal"
},
{
"PeerName": "02:8f:3d:02:22:b2",
"NodeName": "ip-10-80-110-250.ec2.internal"
},
{
"PeerName": "ca:5d:93:ce:43:59",
"NodeName": "ip-10-80-103-14.ec2.internal"
}
]
}I'm not sure how this config map is used by weave, assuming as some sort of state storage, but I expected the number of peers in the annotation to be near to the number of nodes in the cluster, which it wasn't at all. Nodes in cluster at the time was 20 while the number of peers in that list is 17. Below I've attempted to map the peers in the annotation to nodes in the cluster: Above we can see that Secondly there are a few nodes that are missing from the peer list on the annotation, is this a bad thing? I'm going to attempt updating the config map and will report back. Edit: I'm aware of the fix (#3317) added in 2.4.0. We've not upgraded to that version yet, but we will tomorrow. |
|
@itsingoria I opened a new issue #3386 to cover the "unreachable peers" part of your comment which is distinct to the subject of this issue. |
|
@bboreham Thank you. Just a note, Zac and I are on the same team. So if you need anything from us, let us know. |
It's the answer to the question raised at the beginning of #2797 - "How do we even detect that a Weave Net peer originated as a Kubernetes node?". We add each peer to the map on first initialization, and remove it after cleanup.
You can pick one of the missing 3 and look in its logfile at startup to see what happened when it should have added itself. There should be a log line like:
It means they will not be cleaned up when the nodes are deleted from Kubernetes. (I don't believe any of this is particularly related to "attempting to claim same IP range", which uses a different data structure) |
|
Looking at the unreachable peers from #3310 (comment):
The others don't appear at all. Perhaps they were cleaned up on a node the other side of the inconsistency partition, thus the knowledge of the cleanup didn't make it across the whole cluster? |
|
I did not see any logs in response to the request at #3310 (comment). This is still important to discovering the root cause of this issue. Also, I'm not sure if we said here what the cleanup procedure is: To recover, you need to eliminate the IPAM data from the affected nodes and restart. Since you installed via the Kubernetes Addon, this data will be in a file under Since you seem to have about 17 nodes that can talk to each other and 3 that can't talk to them, do this just on the 3. Restarting (rebooting) after removing the file ensures there are no IP addresses in use on the node, which makes the procedure safe. |
|
@bboreham I don't want to celebrate just yet but we've taken the following steps are seeing positive results:
On node deletion I can the see entry remains unreachable on the ipam-list (as I expect) but when a new node comes up, there's cleanup (this wasn't happening before). I want to give it a week and keep checking before I celebrate! |
|
Cool! Please let us know, such that we can also proceed with an update if
appropriate.
…On Thu, Aug 30, 2018, 18:50 itskingori ***@***.***> wrote:
@bboreham <https://github.com/bboreham> I don't want to celebrate just
yet but we've taken the following steps are seeing positive results:
1. Upgraded to 2.4.0 to get the fix @brb <https://github.com/brb>
mentions in #3310 (comment)
<#3310 (comment)>,
then ...
2. Cleared any unreachable! on the ipam-list.
3. Edited the peer-list annotation on the weave-net configmap to make
sure the entries match with the ipam-list (there were entries on the
peer-list that were not on the ipam-list as @zacblazic
<https://github.com/zacblazic> explained in
#3310 (comment)
<#3310 (comment)>
)
4. Cleared state of any pods exhibiting this behaviour ("Received
update for IP range I own") i.e. remove db in /var/lib/weave on host
and delete the pod.
On node deletion I can the see entry remains unreachable on the ipam-list
(as I expect) but when a new node comes up, there's cleanup (this wasn't
happening before).
I want to give it a week and keep checking before I celebrate!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#3310 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AApv1BnA8UcMf0MyWWqnpGaxAW_l-Co3ks5uWBfIgaJpZM4UccSf>
.
|
|
@Raffo if your issue at #3384 is still continuing please post the logs of a |
|
Yes absolutely, I'm keeping an eye on it and will update my issue when I'll have more data. |
|
In #3392 we've found one scenario which leads to the reported error. Could you all run |
|
@brb About 6 days after #3310 (comment), our clusters are still looking healthy. And this is despite scaling up and down quite a bit ... 👇
We can verify that cleanup is taking place ... 👇 kubectl exec -it weave-net-897xt /bin/sh -n kube-system
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-897xt -n kube-system' to see all of the containers in this pod.
76:7f:2a:86:53:7a(ip-10-80-106-202.ec2.internal) 32768 IPs (01.6% of total) (15 active)
6e:0e:51:4f:66:9e(ip-10-80-114-8.ec2.internal) 345486 IPs (16.5% of total)
6a:7d:84:55:63:40(ip-10-80-45-255.ec2.internal) 71362 IPs (03.4% of total)
9a:6b:6f:33:1c:60(ip-10-80-53-166.ec2.internal) 31224 IPs (01.5% of total)
ba:a1:15:9f:64:7b(ip-10-80-113-52.ec2.internal) 16384 IPs (00.8% of total)
c6:51:3e:e0:73:85(ip-10-80-58-36.ec2.internal) 65536 IPs (03.1% of total)
ca:38:9f:69:92:76(ip-10-80-94-19.ec2.internal) 71340 IPs (03.4% of total)
02:8f:3d:02:22:b2(ip-10-80-110-250.ec2.internal) 32776 IPs (01.6% of total)
e2:b7:e3:40:a8:f9(ip-10-80-36-237.ec2.internal) 40872 IPs (01.9% of total)
c2:43:d1:36:2e:44(ip-10-80-46-144.ec2.internal) 528477 IPs (25.2% of total)
ce:81:1f:3d:d1:86(ip-10-80-105-153.ec2.internal) 65536 IPs (03.1% of total)
a2:89:81:f0:f7:cb(ip-10-80-91-69.ec2.internal) 42543 IPs (02.0% of total)
42:90:28:9a:2c:a2(ip-10-80-65-216.ec2.internal) 35616 IPs (01.7% of total)
a2:63:98:df:ee:d0(ip-10-80-41-90.ec2.internal) 27752 IPs (01.3% of total)
72:d3:22:05:8f:08(ip-10-80-112-109.ec2.internal) 143472 IPs (06.8% of total)
3e:8f:f3:15:0c:e6(ip-10-80-67-160.ec2.internal) 131096 IPs (06.3% of total)
be:10:d6:f0:eb:68(ip-10-80-112-120.ec2.internal) 44742 IPs (02.1% of total)
b2:ef:a5:21:f9:a3(ip-10-80-54-76.ec2.internal) 34498 IPs (01.6% of total)
52:ff:e3:6c:74:1d(ip-10-80-66-231.ec2.internal) 24376 IPs (01.2% of total)
52:cb:29:b2:39:4e(ip-10-80-90-159.ec2.internal) 311296 IPs (14.8% of total)We'll continue monitoring. In case it happens again we'll check |
I was facing issues where few nodes were not able to access internet while others were working fine. Deleting /var/lib/weave/weave-netdata.db file and reboot did worked fine but again after 3-4hrs, the issue recreated. I again deleted the file and rebooted and it worked fine again. Why is the file creating issues. Thanks |
|
Linking #1962 which would make this condition less fatal. |
|
I'm going to close this as fixed in 2.6.0 - you may need to reboot a node which is actually using an IP in conflict with another node, but all the un-allocated space should get resolved automatically. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
What you expected to happen?
Nodes should claim their own distinct IP ranges.
What happened?
It seems that two nodes have (or are attempting to) claim the same IP address range.
The node
ip-10-83-42-111.ec2.internalclaimed the100.105.68.0IP range:However, the node
ip-10-83-54-199.ec2.internalis also attempting to claim the100.105.68.0IP range:Restarting the weave pods on either
ip-10-83-42-111.ec2.internalorip-10-83-54-199.ec2.internaldoes not change the situation, likely due to the the fact that state is persisted on disk at/var/lib/weave.During the same time period, there was another issue with weave I noticed. I'm not sure if it is related but some nodes appeared to be attempting to remove a non-existent peer:
According to our logs, this error started on 2018/05/16 (~21 days ago). Around this time we were having issues with weave consuming a large amount of memory.
How to reproduce it?
Not sure if it is easily reproducible, but it occurred while restarting all weave pods in the cluster, with a 15 second sleep between each restart.
There is a possibility that two pods started at the same time due to scheduling delays.
Anything else we need to know?
Cloud provider: AWS, managed by Kops
Versions:
Logs:
Click the black arrows to reveal the details!
From the node with the issue (
ip-10-83-54-199.ec2.internal):Weave IPAM
Weave connections
Weave logs
From a node that has already claimed the IP range (
ip-10-83-42-111.ec2.internal):Weave IPAM
Weave connections
Weave logs
The text was updated successfully, but these errors were encountered: