"Sometimes containers cannot connect but they still respond to ping" #2433
Comments
|
The error message is the same as that noted in #2297. |
|
It seems that we can reproduce this pretty easily with 5 node cluster by scaling one service ~20 times up/down (between 0-100 containers). Weave: 1.4.5 |
|
@bboreham was pondering whether we can make a better decision by inspecting the destination mac, i.e. only drop the mac if the source mac appears to be from a different peer and the destination mac is for a local peer. Which, by my reading of the code, simply means turning the error into a warning and carrying on. We'd need to think carefully in what situations that might create a loop. btw, afaict, we'd get this error if a MAC moved or was duplicated. Correct? Surely we ought to handle the former. |
We have code in the handler of forwarded packets for that, but not captured packets. Surely we need both. Without the latter when a MAC moves from another peer to ourselves, we will be suppressing all outgoing packets from that MAC, won't we? |
Reproduced and filed #2436. |
|
Hi gents, downstream user of Weave. We're experiencing this issue and happy to help in any way we can. |
|
Hi @crockpotveggies - thanks. I have a specific question - are you taking any steps outside of weave to adjust or set container MACs, e.g. for the purposes of migration? The MACs of weave's |
|
@jakolehm Could you provide steps for reproducing the issue? Thanks. |
|
@jakolehm @crockpotveggies next time you get the on the affected host and see if you have |
|
Sorry late reply here, fairly busy on my end. Regarding "taking any steps outside of weave" I can't fully answer the question because, as a downstream user, we rely on what the Docker Cloud agent does for us. The circumstances leading to failure involve redeploys within our infrastructure. Basically, we "reload" our backend services (which involves around 10 containers to reproduce this issue) and suddenly we start seeing timeouts to our RabbitMQ container and our main database. When we open up RabbitMQ to a public address and bypass weave so the containers connect to a public IP directly, it completely alleviates the problem...which, of course, opens us up to more shenanigans. |
|
@awh yes it has hairpin: |
|
bingo! any idea how it ended up like that? |
|
no idea, where should I look? :) |
|
Oh, sorry for confusion.. hairpin is off in that output :/ |
|
bah, that output format is terrible. |
|
Thanks @jakolehm - back to the drawing board then! We'd like to try to replicate what you mentioned earlier:
Could you answer a few more questions?
Essentially as much concrete information as you are able to give about the setup and testing methodology - if there are images and configs you can share so much the better. Thanks! |
We have been trying to reproduce this locally without success :( |
|
Great detail, thanks!
That's interesting... what're you using locally? |
|
Exactly same stuff but running inside vagrant boxes. |
|
Hi gents, I've been doing packet captures, here's what's coming from
|
|
Additionally, here's what
|
|
I may have identified a culprit. I checked the logs in the router that handles all of our offline processing nodes and found that it seems to think it's under constant attack. Is it possible that Weave is triggering security systems in networks meant to protect against DoS attacks?
|
|
Thanks @crockpotveggies - let us digest that. Out of interest, are you using sleeve as well? |
|
@jakolehm @crockpotveggies is there any possibility we could gain interactive access to one of your affected systems when this occurs again? |
|
@awh negative not using sleeve (unless Docker Cloud has implemented it upstream). FYI the IP address shown in the logs is the IP of the router that handles our small on-site datacenter, meaning it came from within. I'm happy to grant access if it happens again, though I have a feeling we won't. Literally hacked a special version of router software just to accommodate this special case. |
|
Note that #2674, released in Weave Net 1.8.2, added more checks and logging which will be useful if this reoccurs. |
|
I debugged the original issue with the GCE cluster (running Weave 1.5.2) in-situ, and this is actually something closer to #1455, with zombie network namespaces that get left behind after the Docker container itself is removed, that still have an ARP from the client containers returns one of two ethernet addresses. One of those is associated with the "broken" container that is running the service we're attempt to connecting to, but the other ethernet address can be found on a different host, where the same Weave IP address was used by a container that is no longer running, and the overlay IP has been re-allocated for use by the newer container. In this case, this zombie container was started immediately after the initial host reboot, and terminated approximately a week before debugging this issue. TCP connections fail with RST -> ECONNREFUSED because the server app process is no longer running in that netns, but ARP and ICMP ping still work because the netns is alive in the kernel. The symptom of this is a host-side Unforunately while I've been able to track down these zombie veths and verified that unlinking them from the |
|
@SpComb are you running cadvisor? |
Yes, but I IIRC dug through the I can take a better dig at those next time I run across such a zombie to dissect. |
|
@SpComb kernel version? |
|
I'm not sure I noted that down, but it would have been CoreOS stable as of October, so likely Linux 4.7.3 |
|
I am experiencing this problem on my Kubernetes cluster. Here are some details: Kubernetes Version: v1.4.4 |
|
@Starefossen what does |
|
I'm probably having the same issue. docker: Docker version 1.12.5, build 047e51b/1.12.5 |
|
@panuhorsmalahti "the same issue" being "seeing the same MAC as coming from two different peers" ? Can you show some of the evidence that leads you to that thought? |
|
I can ping and curl the container IP from the host, and can ping the container from another container, but curl doesn't work. weave-zombie-hunter.sh output: arping output: Not 100% sure this is the same issue, was linked here by Kontena slack. |
|
@panuhorsmalahti unfortunately we have two sets of symptoms in this issue; please can you open your own issue to avoid making things worse? |
|
@panuhorsmalahti I read carefully through the output you supplied from The way I read the script, if it prints Also, it seems to be saying that every single one of the veths on your system is "nomaster". Are you running the same scripts I am reading? |
|
Created a new issue: #2842 |
Seems similar to another report on Slack from @fermayo; TL;DR is that Weave is apparently seeing the same MAC as coming from two different peers, and this happens at the same time as networking gets broken to the container that really owns that MAC.
I tried to recreate locally and failed.
As reported on weave-users Slack:
jakolehm "we have really strange problem with weave net.. sometimes containers cannot connect but they still respond to ping (weave 1.4.5)
and this happens on coreos stable (latest)
and this happens pretty randomly
let's say we have two peers, A and B ... if we send request from peer A to container in peer B we don't see packet in peer B at all
bryan do you see it on the weave bridge?
jakolehm no
only on peer A weave bridge
bryan ok, this is crossing two hosts, so you have two weave bridges (edited)
jakolehm yes
ping seems to work, and we see the packets
bryan Check the destination MAC address does actually correspond to the container you are trying to hit
jakolehm it does, checked
but, ping reply mac was different
jakolehm actually ping reply mac address is something that we cannot find in any of the machines in this cluster
jakolehm actually it seems that request destination mac is wrong also for tcp connections
ba:6e:b6:d8:d8:b9should bef2:e2:6e:f4:a3:cematthias what host does weave think that MAC is on? logs should tell you.
jakolehm
jakolehm bryan: what is a "local mac" ... where does weave get that?
bryan It's printed when we see a packet with a source mac we have never seen before, on the weave bridge.
Since there ought to be no way for packets to get onto the bridge except from a locally-running container, we think it's from one of those.
jakolehm but I can't find that mac on "gcp-1-4" machine
bryan it's possible it went away
jakolehm but I restarted weave and it's coming back...
bryan that's interesting
jakolehm one of the first locally discovered macs
bryan I guess you could
tcpdumpthe weave bridge and see if the packet itself gives any cluesthis is somewhat consistent with the "MAC associated with another peer" message - if we've never seen the src address before we print "local MAC", and if we have seen it on another peer we print "associated ..."
so, since you do get the latter, it must be something of a race which one is taken as the "real" home of the packet
and the real question is how come we are seeing packets with the same src address on two different weave bridges?
jakolehm
bryan and is that container on a different machine?
jakolehm yes
The text was updated successfully, but these errors were encountered: