iptables: No chain/target/match by that name.

[errordeveloper]

I've installed weave-kube on a Kubernetes cluster, after rebooting a node, weave-kube fails to start and logs this:

root@kube-node-0:~# docker logs cfecf98444b0                                                              
INFO: 2016/10/24 17:22:36.316253 Command line options: map[ipalloc-range:10.32.0.0/12 name:ca:dd:16:be:df:
42 nickname:kube-node-0 no-dns:true docker-api: datapath:datapath http-addr:127.0.0.1:6784 ipalloc-init:co
nsensus=2 port:6783]                                                                                      
INFO: 2016/10/24 17:22:36.339086 Communication between peers is unencrypted.                              
INFO: 2016/10/24 17:22:36.342394 Our name is ca:dd:16:be:df:42(kube-node-0)                               
INFO: 2016/10/24 17:22:36.342490 Launch detected - using supplied peer list: [163.172.63.141 62.210.116.20
6]                                                                                                        
INFO: 2016/10/24 17:22:36.406053 [allocator ca:dd:16:be:df:42] Initialising with persisted data           
INFO: 2016/10/24 17:22:36.406158 Sniffing traffic on datapath (via ODP)                                   
INFO: 2016/10/24 17:22:36.407127 ->[62.210.116.206:6783] attempting connection                            
INFO: 2016/10/24 17:22:36.407365 ->[163.172.63.141:6783] attempting connection                            
INFO: 2016/10/24 17:22:36.407662 ->[163.172.63.141:57223] connection accepted                             
INFO: 2016/10/24 17:22:36.411021 ->[62.210.116.206:6783|3e:62:bc:39:9d:42(kube-node-1)]: connection ready;
 using protocol version 2                                                                                 
INFO: 2016/10/24 17:22:36.411151 overlay_switch ->[3e:62:bc:39:9d:42(kube-node-1)] using fastdp           
INFO: 2016/10/24 17:22:36.411262 ->[62.210.116.206:6783|3e:62:bc:39:9d:42(kube-node-1)]: connection added 
(new peer)                                                                                                
INFO: 2016/10/24 17:22:36.413027 Listening for HTTP control messages on 127.0.0.1:6784                    
INFO: 2016/10/24 17:22:36.414151 ->[163.172.63.141:6783|ca:dd:16:be:df:42(kube-node-0)]: connection shutti
ng down due to error: cannot connect to ourself                                                           
INFO: 2016/10/24 17:22:36.414268 ->[163.172.63.141:57223|ca:dd:16:be:df:42(kube-node-0)]: connection shutt
ing down due to error: cannot connect to ourself                                                          
INFO: 2016/10/24 17:22:36.912987 ->[62.210.116.206:6783|3e:62:bc:39:9d:42(kube-node-1)]: connection fully 
established                                                                                               
INFO: 2016/10/24 17:22:36.913654 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, paylo
ad was 60020 bytes)                                                                                       
INFO: 2016/10/24 17:22:36.914948 sleeve ->[62.210.116.206:6783|3e:62:bc:39:9d:42(kube-node-1)]: Effective 
MTU verified at 1438                                                                                      
iptables: No chain/target/match by that name.                                                             

[errordeveloper]

The weave-npc container runs fine, just the weave container is failing.

[errordeveloper]

I've tried to re-create the addon, but nothing changed.

[errordeveloper]

I've been able to get around this by deleting the addon, running weave reset on the node, and re-creating the addon.

[bboreham]

The symptoms all match this message coming from weave expose setting up the NAT masquerading rules. And since the nat table existed the second time you did it, this points toward the WEAVE chain being missing. Since we don't have the error message from that run, weave reset (or delete the bridge some other way) and re-try is the appropriate action.

We could certainly make the log messages easier to tie to what was happening.

Since Kubernetes effectively buries the container log messages by deleting the container and re-trying, it's difficult to diagnose fully, and also difficult for the end-user to cure a part-successful launch. Can we do better?

[awh]

@errordeveloper could we get information on k8s/distro versions, cloud provider etc? The evidence points to something external to weave-kube having removed the WEAVE chain from the nat table partway through launch - it'd be good to know what else was running at the time (e.g. firewalld)

[raghu67]

I am seeing a similar issue. Here are the details:
[root@m0062421 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:16:57Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:10:32Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
[root@m0062421 ~]# kubeadm version
kubeadm version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.0.1534+cf7301f16c0363-dirty", GitCommit:"cf7301f16c036363c4fdcb5d4d0c867720214598", GitTreeState:"dirty", BuildDate:"2016-09-27T18:10:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
[root@m0062421 ~]#

CentOS 7.2. Kernel Version:
[root@m0062421 ~]# uname -a
Linux m0062421.lab.ppops.net 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

These are based on an old version of OpenStack and KVM. if that is relevant, I can find the details

[awh]

This issue was moved to weaveworks/weave#2617