Skip to content
This repository has been archived by the owner on Jun 20, 2024. It is now read-only.

kube-weave weave-npc failing to start #2653

Closed
chrislovecnm opened this issue Nov 17, 2016 · 10 comments
Closed

kube-weave weave-npc failing to start #2653

chrislovecnm opened this issue Nov 17, 2016 · 10 comments
Assignees
@awh
Labels
bug
Milestone
1.9.0

Comments

@chrislovecnm
Copy link

chrislovecnm commented Nov 17, 2016 •
edited
Loading

Hi there

I am launch a kops weave cluster with private networking, @bboreham knows what I mean. We have 20 nodes, and on one of them weave-npc is failing to start. The issue is that ulogd is failing with:

 # ulogd -v
Thu Nov 17 21:21:30 2016 <5> ulogd.c:843 building new pluginstance stack: 'log1:NFLOG,base1:BASE,pcap1:PCAP'
Thu Nov 17 21:21:30 2016 <7> ulogd_inppkt_NFLOG.c:552 unable to bind to log group 86
Thu Nov 17 21:21:30 2016 <7> ulogd.c:813 error starting `log1'
Thu Nov 17 21:21:30 2016 <8> ulogd.c:1430 not even a single working plugin stack

I have launched a pod onto the impacted node with a nodeSelector and am running from a single weave-npc pod.

Here is a full strace for ulogd

/ # strace -s 1024 ulogd -v
execve("/usr/sbin/ulogd", ["ulogd", "-v"], [/* 26 vars */]) = 0
arch_prctl(ARCH_SET_FS, 0x7f65bd3adb28) = 0
set_tid_address(0x7f65bd3adb60)         = 25878
mprotect(0x7f65bd3aa000, 4096, PROT_READ) = 0
mprotect(0x55d722f3a000, 4096, PROT_READ) = 0
access("/etc/ulogd.conf", R_OK)         = 0
open("/etc/ulogd.conf", O_RDONLY)       = 3
readv(3, [{"", 0}, {"[global]\nlogfile=\"/dev/null\"\nplugin=\"/usr/lib/ulogd/ulogd_inppkt_NFLOG.so\"\nplugin=\"/usr/lib/ulogd/ulogd_raw2packet_BASE.so\"\nplugin=\"/usr/lib/ulogd/ulogd_output_PCAP.so\"\nstack=log1:NFLOG,base1:BASE,pcap1:PCAP\n\n[log1]\ngroup=86\n\n[pcap1]\nfile=\"/var/log/ulogd.pcap\"\nsync=1\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 1024}], 2) = 268
open("/dev/null", O_WRONLY|O_CREAT|O_APPEND, 0666) = 4
fcntl(4, F_GETFL)                       = 0x8401 (flags O_WRONLY|O_APPEND|O_LARGEFILE)
ioctl(4, TIOCGWINSZ, 0x7ffe7ee9f108)    = -1 ENOTTY (Not a tty)
open("/usr/lib/ulogd/ulogd_inppkt_NFLOG.so", O_RDONLY|O_CLOEXEC) = 5
fcntl(5, F_SETFD, FD_CLOEXEC)           = 0
fstat(5, {st_mode=S_IFREG|0755, st_size=18816, ...}) = 0
read(5, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \22\0\0\0\0\0\0@\0\0\0\0\0\0\0\300C\0\0\0\0\0\0\0\0\0\0@\0008\0\6\0@\0\27\0\26\0\1\0\0\0\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0|\"\0\0\0\0\0\0|\"\0\0\0\0\0\0\0\0 \0\0\0\0\0\1\0\0\0\6\0\0\0\230,\0\0\0\0\0\0\230, \0\0\0\0\0\230, \0\0\0\0\0`\26\0\0\0\0\0\0\270\26\0\0\0\0\0\0\0\0 \0\0\0\0\0\2\0\0\0\6\0\0\0\310,\0\0\0\0\0\0\310, \0\0\0\0\0\310, \0\0\0\0\0\220\1\0\0\0\0\0\0\220\1\0\0\0\0\0\0\10\0\0\0\0\0\0\0P\345td\4\0\0\0@ \0\0\0\0\0\0@ \0\0\0\0\0\0@ \0\0\0\0\0\0\\\0\0\0\0\0\0\0\\\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0Q\345td\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0R\345td\4\0\0\0\230,\0\0\0\0\0\0\230, \0\0\0\0\0\230, \0\0\0\0\0h\3\0\0\0\0\0\0h\3\0\0\0\0\0\0\1\0\0\0\0\0\0\0\3\0\0\0003\0\0\0\1\0\0\0\6\0\0\0\212\3000\1\0\6@\v3\0\0\0006\0\0\0009\0\0\0@:\353,BE\325\354\273\343\222|\270\215\361\0168\205\230|\331qX\34\353\323\357\16\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\3\0\5\0\210\20\0\0\0\0\0\0\0\0\0\0\0\0\0\0003\2\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\260\2\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0=\1\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\314\0\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\302\2\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\374\2\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\27\3\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0c\2\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\v\1\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\351\1\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\220\2\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0-\3\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\376\1\0\0\20\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\324\2\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0o\2\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0C\0\0\0\"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\250\0\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0~\2\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0Q\2\0\0\20\0\0\0\0\0\0\0\0\0\0\0", 960) = 960
mmap(NULL, 2117632, PROT_READ|PROT_EXEC, MAP_PRIVATE, 5, 0) = 0x7f65bcf1e000
mmap(0x7f65bd120000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 5, 0x2000) = 0x7f65bd120000
close(5)                                = 0
open("/etc/ld-musl-x86_64.path", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib/libnetfilter_log.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libnetfilter_log.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libnetfilter_log.so.1", O_RDONLY|O_CLOEXEC) = 5
fcntl(5, F_SETFD, FD_CLOEXEC)           = 0
fstat(5, {st_mode=S_IFREG|0755, st_size=13560, ...}) = 0
read(5, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0x\21\0\0\0\0\0\0@\0\0\0\0\0\0\0\2700\0\0\0\0\0\0\0\0\0\0@\0008\0\6\0@\0\21\0\20\0\1\0\0\0\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\320(\0\0\0\0\0\0\320(\0\0\0\0\0\0\0\0 \0\0\0\0\0\1\0\0\0\6\0\0\0\210-\0\0\0\0\0\0\210- \0\0\0\0\0\210- \0\0\0\0\0\220\2\0\0\0\0\0\0\230\2\0\0\0\0\0\0\0\0 \0\0\0\0\0\2\0\0\0\6\0\0\0\210-\0\0\0\0\0\0\210- \0\0\0\0\0\210- \0\0\0\0\0@\1\0\0\0\0\0\0@\1\0\0\0\0\0\0\10\0\0\0\0\0\0\0P\345td\4\0\0\0(\"\0\0\0\0\0\0(\"\0\0\0\0\0\0(\"\0\0\0\0\0\0<\1\0\0\0\0\0\0<\1\0\0\0\0\0\0\4\0\0\0\0\0\0\0Q\345td\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0R\345td\4\0\0\0\210-\0\0\0\0\0\0\210- \0\0\0\0\0\210- \0\0\0\0\0x\2\0\0\0\0\0\0x\2\0\0\0\0\0\0\1\0\0\0\0\0\0\0%\0\0\0\32\0\0\0\4\0\0\0\10\0\0\0A\20Hk\326\200\4H(\10\0\10\230\22\5\t\1\"U\"\30\247=\f\0\0\22\5\200\4\222@\32\0\0\0\0\0\0\0\0\0\0\0\33\0\0\0\0\0\0\0\34\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\37\0\0\0 \0\0\0!\0\0\0#\0\0\0&\0\0\0*\0\0\0,\0\0\0\0\0\0\0\0\0\0\0-\0\0\0\0\0\0\0.\0\0\0001\0\0\0002\0\0\0004\0\0\0005\0\0\0\0\0\0\0\0\0\0\0006\0\0\0\0\0\0\0008\0\0\0:\0\0\0\0\0\0\0<\0\0\0=\0\0\0\0\0\0\0>\0\0\0\rg\261\v\353\232\0\f\222\211F\337.\276\365\25\23VU\220cgbh\261T\311H\"\35\252\2033;L\1\214\352\251\203\376\2645l\325w\206\251p\3433\343\346\221J\345xi\313\337\273R\4\323\220\226\5H\33&\252\203\r\241L_\245\250\326 \230RL\200l$p\356%\330\260\257\263\300\252\304\2645\274\244\257\331\30j\331qX\34\273\343\222|\6Y\313\347\263\300\26\346\256\243$qCE\325\354r\233\223\213\255\20\336\222C;\370qK\2332e\234\3517\240\1\36-H\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\3\0\7\0x\21\0\0\0\0\0\0\0\0\0\0\0\0\0\0\324\0\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0n\3\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\257\0\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\331\1\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\376\1\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0m\0\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\17\0\0\0\22\0\0\0\0\0\0\0\0\0\0\0", 960) = 960
mmap(NULL, 2113536, PROT_READ|PROT_EXEC, MAP_PRIVATE, 5, 0) = 0x7f65bcd1a000
mmap(0x7f65bcf1c000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 5, 0x2000) = 0x7f65bcf1c000
close(5)                                = 0
open("/lib/libnfnetlink.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libnfnetlink.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libnfnetlink.so.0", O_RDONLY|O_CLOEXEC) = 5
fcntl(5, F_SETFD, FD_CLOEXEC)           = 0
fstat(5, {st_mode=S_IFREG|0755, st_size=25680, ...}) = 0
read(5, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\26\0\0\0\0\0\0@\0\0\0\0\0\0\0\220`\0\0\0\0\0\0\0\0\0\0@\0008\0\6\0@\0\17\0\16\0\1\0\0\0\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\330R\0\0\0\0\0\0\330R\0\0\0\0\0\0\0\0 \0\0\0\0\0\1\0\0\0\6\0\0\0P]\0\0\0\0\0\0P] \0\0\0\0\0P] \0\0\0\0\0\260\2\0\0\0\0\0\0\260\2\0\0\0\0\0\0\0\0 \0\0\0\0\0\2\0\0\0\6\0\0\0P]\0\0\0\0\0\0P] \0\0\0\0\0P] \0\0\0\0\0000\1\0\0\0\0\0\0000\1\0\0\0\0\0\0\10\0\0\0\0\0\0\0P\345td\4\0\0\0\220E\0\0\0\0\0\0\220E\0\0\0\0\0\0\220E\0\0\0\0\0\0\374\1\0\0\0\0\0\0\374\1\0\0\0\0\0\0\4\0\0\0\0\0\0\0Q\345td\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0R\345td\4\0\0\0P]\0\0\0\0\0\0P] \0\0\0\0\0P] \0\0\0\0\0\260\2\0\0\0\0\0\0\260\2\0\0\0\0\0\0\1\0\0\0\0\0\0\0%\0\0\0\35\0\0\0\10\0\0\0\t\0\0\0\2\1\v\204\240\211\2 \20\31\240d\"\200\20\\\0\0\0\26\312\2\2\f\0\4@\f8\0 \0\310 \3\210\202D\2\20\232\4AA\25\t\340\30\f\0\1\200$\1B\211\24\7\5\211\f\1\0\1\35\0\0\0\36\0\0\0 \0\0\0\"\0\0\0\0\0\0\0\0\0\0\0$\0\0\0&\0\0\0)\0\0\0+\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0-\0\0\0\0\0\0\0\0\0\0\0.\0\0\0000\0\0\0001\0\0\0003\0\0\0005\0\0\0008\0\0\0009\0\0\0<\0\0\0?\0\0\0B\0\0\0F\0\0\0G\0\0\0J\0\0\0K\0\0\0N\0\0\0O\0\0\0R\0\0\0S\0\0\0U\0\0\0V\0\0\0_\210;8j\353\23A1\220\370Y<\r;8\327\3104\214\32N\2465\311\25YT\220E\307\222=\264\315V\350\t\177>\364\24y>\3436E\217\344J\353,{=#\177\236\7\36\253}J7\rA\241*#*\2\266\"\203\177:8\2738\256\25T\226\370YW\301\344\375\6\217\213\10\321E\27\377\20&=\216\2\33C\216\261v\24\321/\"W!LhE\27H\256\370m\37\2\306\354`\272\315V\266q\304\365\331qX\34$P\352{\272\343\222|\37\343\3412n\220\370Yv\321C-\246\302]@1J\23\322\251K\2263\3367?\22\242\262\236\231\303G68/\317S\314X\0179\307\310%\201?CE\325\354{\264\315V*\241\236\231J\353\343\303\33\243\320\367\3654;\361\246\264\10\4'C\275Z{|w\255\242\322\33\323d\t98wy\330\32\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\3\0\7\0\20\26\0\0\0\0\0\0\0\0\0\0\0\0\0\0\212\3\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\261\1\0\0\22\0\0\0\0\0\0\0\0\0\0\0", 960) = 960
mmap(NULL, 2121728, PROT_READ|PROT_EXEC, MAP_PRIVATE, 5, 0) = 0x7f65bcb14000
mmap(0x7f65bcd19000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 5, 0x5000) = 0x7f65bcd19000
close(5)                                = 0
mprotect(0x7f65bd120000, 4096, PROT_READ) = 0
mprotect(0x7f65bcf1c000, 4096, PROT_READ) = 0
mprotect(0x7f65bcd19000, 4096, PROT_READ) = 0
open("/usr/lib/ulogd/ulogd_raw2packet_BASE.so", O_RDONLY|O_CLOEXEC) = 5
fcntl(5, F_SETFD, FD_CLOEXEC)           = 0
fstat(5, {st_mode=S_IFREG|0755, st_size=14808, ...}) = 0
read(5, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\370\6\0\0\0\0\0\0@\0\0\0\0\0\0\0\0304\0\0\0\0\0\0\0\0\0\0@\0008\0\6\0@\0\27\0\26\0\1\0\0\0\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\364\23\0\0\0\0\0\0\364\23\0\0\0\0\0\0\0\0 \0\0\0\0\0\1\0\0\0\6\0\0\0\370\35\0\0\0\0\0\0\370\35 \0\0\0\0\0\370\35 \0\0\0\0\0X\25\0\0\0\0\0\0\270\25\0\0\0\0\0\0\0\0 \0\0\0\0\0\2\0\0\0\6\0\0\0(\36\0\0\0\0\0\0(\36 \0\0\0\0\0(\36 \0\0\0\0\0p\1\0\0\0\0\0\0p\1\0\0\0\0\0\0\10\0\0\0\0\0\0\0P\345td\4\0\0\0\364\21\0\0\0\0\0\0\364\21\0\0\0\0\0\0\364\21\0\0\0\0\0\0L\0\0\0\0\0\0\0L\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0Q\345td\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0R\345td\4\0\0\0\370\35\0\0\0\0\0\0\370\35 \0\0\0\0\0\370\35 \0\0\0\0\0\10\2\0\0\0\0\0\0\10\2\0\0\0\0\0\0\1\0\0\0\0\0\0\0\3\0\0\0\f\0\0\0\1\0\0\0\6\0\0\0\210\3000\1\0\4@\v\f\0\0\0\16\0\0\0\21\0\0\0BE\325\354\273\343\222|\270\215\361\0168\205\230|\331qX\34\353\323\357\16\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\3\0\5\0\230\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\224\0\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0C\0\0\0\"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\241\0\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0R\0\0\0 \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0)\0\0\0 \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\r\0\0\0 \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\233\0\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\247\0\0\0\20\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\200\0\0\0 \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0j\0\0\0 \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\323\0\0\0\20\0\23\0P3 \0\0\0\0\0\0\0\0\0\0\0\0\0\346\0\0\0\20\0\24\0\2603 \0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\22\0\5\0\230\6\0\0\0\0\0\0\1\0\0\0\0\0\0\0\2\0\0\0\22\0\10\0\370\6\0\0\0\0\0\0\f\0\0\0\0\0\0\0\332\0\0\0\20\0\24\0P3 \0\0\0\0\0\0\0\0\0\0\0\0\0\7\0\0\0\22\0\t\0\232\21\0\0\0\0\0\0\1\0\0\0\0\0\0\0\0_init\0_fini\0_ITM_deregisterTMCloneTable\0_ITM_registerTMCloneTab", 960) = 960
mmap(NULL, 2113536, PROT_READ|PROT_EXEC, MAP_PRIVATE, 5, 0) = 0x7f65bc910000
mmap(0x7f65bcb11000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 5, 0x1000) = 0x7f65bcb11000
close(5)                                = 0
mprotect(0x7f65bcb11000, 4096, PROT_READ) = 0
open("/usr/lib/ulogd/ulogd_output_PCAP.so", O_RDONLY|O_CLOEXEC) = 5
fcntl(5, F_SETFD, FD_CLOEXEC)           = 0
fstat(5, {st_mode=S_IFREG|0755, st_size=11104, ...}) = 0
read(5, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0H\t\0\0\0\0\0\0@\0\0\0\0\0\0\0\240%\0\0\0\0\0\0\0\0\0\0@\0008\0\6\0@\0\27\0\26\0\1\0\0\0\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\f\20\0\0\0\0\0\0\f\20\0\0\0\0\0\0\0\0 \0\0\0\0\0\1\0\0\0\6\0\0\0\240\35\0\0\0\0\0\0\240\35 \0\0\0\0\0\240\35 \0\0\0\0\0008\7\0\0\0\0\0\0\220\7\0\0\0\0\0\0\0\0 \0\0\0\0\0\2\0\0\0\6\0\0\0\320\35\0\0\0\0\0\0\320\35 \0\0\0\0\0\320\35 \0\0\0\0\0\200\1\0\0\0\0\0\0\200\1\0\0\0\0\0\0\10\0\0\0\0\0\0\0P\345td\4\0\0\0\234\16\0\0\0\0\0\0\234\16\0\0\0\0\0\0\234\16\0\0\0\0\0\0L\0\0\0\0\0\0\0L\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0Q\345td\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0R\345td\4\0\0\0\240\35\0\0\0\0\0\0\240\35 \0\0\0\0\0\240\35 \0\0\0\0\0`\2\0\0\0\0\0\0`\2\0\0\0\0\0\0\1\0\0\0\0\0\0\0\3\0\0\0\26\0\0\0\1\0\0\0\6\0\0\0\210\3000\1\0\4@\v\26\0\0\0\30\0\0\0\33\0\0\0BE\325\354\273\343\222|\270\215\361\0168\205\230|\331qX\34\353\323\357\16\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\3\0\5\0\250\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\267\0\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0C\0\0\0\"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\353\0\0\0\20\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\344\0\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\314\0\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\300\0\0\0\20\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0R\0\0\0 \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0)\0\0\0 \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0E\1\0\0\21\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\r\0\0\0 \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\375\0\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\240\0\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\224\0\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\n\1\0\0\20\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0C\1\0\0\21\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\233\0\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\335\0\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\246\0\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\200\0\0\0 \0\0\0\0\0\0\0\0\0\0\0", 960) = 960
mmap(NULL, 2109440, PROT_READ|PROT_EXEC, MAP_PRIVATE, 5, 0) = 0x7f65bc70d000
mmap(0x7f65bc90e000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 5, 0x1000) = 0x7f65bc90e000
close(5)                                = 0
open("/lib/libpcap.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libpcap.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libpcap.so.1", O_RDONLY|O_CLOEXEC) = 5
fcntl(5, F_SETFD, FD_CLOEXEC)           = 0
fstat(5, {st_mode=S_IFREG|0755, st_size=235712, ...}) = 0
read(5, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\350x\0\0\0\0\0\0@\0\0\0\0\0\0\0\0\223\3\0\0\0\0\0\0\0\0\0@\0008\0\6\0@\0\27\0\26\0\1\0\0\0\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0lk\3\0\0\0\0\0lk\3\0\0\0\0\0\0\0 \0\0\0\0\0\1\0\0\0\6\0\0\0\330u\3\0\0\0\0\0\330u#\0\0\0\0\0\330u#\0\0\0\0\0b\34\0\0\0\0\0\0\250)\0\0\0\0\0\0\0\0 \0\0\0\0\0\2\0\0\0\6\0\0\0\240\206\3\0\0\0\0\0\240\206#\0\0\0\0\0\240\206#\0\0\0\0\0P\1\0\0\0\0\0\0P\1\0\0\0\0\0\0\10\0\0\0\0\0\0\0P\345td\4\0\0\0\370\35\3\0\0\0\0\0\370\35\3\0\0\0\0\0\370\35\3\0\0\0\0\0\224\v\0\0\0\0\0\0\224\v\0\0\0\0\0\0\4\0\0\0\0\0\0\0Q\345td\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0R\345td\4\0\0\0\330u\3\0\0\0\0\0\330u#\0\0\0\0\0\330u#\0\0\0\0\0(\32\0\0\0\0\0\0(\32\0\0\0\0\0\0\1\0\0\0\0\0\0\0\305\0\0\0c\0\0\0 \0\0\0\v\0\0\0\242\n\0\300I\10\202\tA\26\204\24\202\215 \0\4\0\0\0 \220\10\24\260\0|\2\2\tp\4\31\0\0\0\0\30\2\0\302\34\30\301\204\10\322\24\0\0\0\0\10@\0\0\0\300\f\1@\10! \4( \201C\210\10\0\10b\0@P\201&D d\21\0%\225\r\0\30\240\10\200\1B(\10\f \202\0\0\0\0\21P\1\4DHHHC8\206!\20\1(\207X\6\0\302 \200\4\30\26\200\0\271\0`@!\234\222\1\311\220M\0\241v\0060D\201\0\21 \341`\6\2\4\7\t\1\220\0a\0$\20\0210@\10 \220$\244Q\10\250\0\10\20\0\0\0\2\0010\2\2244\4\4 \0\250\4\nH\201\203\0\0\300\nB\5d\20\210\ft0\0\0\2\350 \0\24D\0\10\10\20\4DIQ \344\0 \204\35\30\4\2AA`\207\20\7\10\2`$\20\2\3\3\4\0\1\211\2402$c\0\0\0\0\0\0\0e\0\0\0\0\0\0\0g\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0h\0\0\0k\0\0\0m\0\0\0p\0\0\0q\0\0\0r\0\0\0t\0\0\0w\0\0\0x\0\0\0|\0\0\0~\0\0\0\0\0\0\0\177\0\0\0\200\0\0\0\202\0\0\0\204\0\0\0\205\0\0\0\0\0\0\0\210\0\0\0\211\0\0\0\213\0\0\0\0\0\0\0\214\0\0\0\0\0\0\0\216\0\0\0\0\0\0\0\220\0\0\0\0\0\0\0\224\0\0\0\226\0\0\0\230\0\0\0\231\0\0\0\235\0\0\0\0\0\0\0\237\0\0\0\0\0\0\0\241\0\0\0\242\0\0\0\245\0\0\0\246\0\0\0\247\0\0\0\250\0\0\0\251\0\0\0\253\0\0\0\254\0\0\0\256\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\260\0\0\0\262\0\0\0\264\0\0\0\265\0\0\0\270\0\0\0\271\0\0\0\272\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\273\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\276\0\0\0", 960) = 960
mmap(NULL, 2334720, PROT_READ|PROT_EXEC, MAP_PRIVATE, 5, 0) = 0x7f65bc4d3000
mmap(0x7f65bc70a000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 5, 0x37000) = 0x7f65bc70a000
close(5)                                = 0
mprotect(0x7f65bc90e000, 4096, PROT_READ) = 0
mprotect(0x7f65bc70a000, 8192, PROT_READ) = 0
clock_gettime(CLOCK_REALTIME, {1479417537, 496274100}) = 0
open("/etc/localtime", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = 5
fstat(5, {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
mmap(NULL, 118, PROT_READ, MAP_SHARED, 5, 0) = 0x7f65bd3a9000
close(5)                                = 0
writev(2, [{"Thu Nov 17 21:18:57 2016 <5> ulogd.c:843 ", 41}, {NULL, 0}], 2Thu Nov 17 21:18:57 2016 <5> ulogd.c:843 ) = 41
writev(4, [{"Thu Nov 17 21:18:57 2016 <5> ulogd.c:843 building new pluginstance stack: 'log1:NFLOG,base1:BASE,pcap1:PCAP'\n", 109}, {NULL, 0}], 2) = 109
writev(2, [{"building new pluginstance stack: 'log1:NFLOG,base1:BASE,pcap1:PCAP'\n", 68}, {NULL, 0}], 2building new pluginstance stack: 'log1:NFLOG,base1:BASE,pcap1:PCAP'
) = 68
brk(NULL)                               = 0x55d724d09000
brk(0x55d724d0b000)                     = 0x55d724d0b000
brk(0x55d724d0d000)                     = 0x55d724d0d000
open("/etc/ulogd.conf", O_RDONLY)       = 5
readv(5, [{"", 0}, {"[global]\nlogfile=\"/dev/null\"\nplugin=\"/usr/lib/ulogd/ulogd_inppkt_NFLOG.so\"\nplugin=\"/usr/lib/ulogd/ulogd_raw2packet_BASE.so\"\nplugin=\"/usr/lib/ulogd/ulogd_output_PCAP.so\"\nstack=log1:NFLOG,base1:BASE,pcap1:PCAP\n\n[log1]\ngroup=86\n\n[pcap1]\nfile=\"/var/log/ulogd.pcap\"\nsync=1\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 1024}], 2) = 268
readv(5, [{"", 0}, {"[global]\nlogfile=\"/dev/null\"\nplugin=\"/usr/lib/ulogd/ulogd_inppkt_NFLOG.so\"\nplugin=\"/usr/lib/ulogd/ulogd_raw2packet_BASE.so\"\nplugin=\"/usr/lib/ulogd/ulogd_output_PCAP.so\"\nstack=log1:NFLOG,base1:BASE,pcap1:PCAP\n\n[log1]\ngroup=86\n\n[pcap1]\nfile=\"/var/log/ulogd.pcap\"\nsync=1\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 1024}], 2) = 0
close(5)                                = 0
open("/etc/ulogd.conf", O_RDONLY)       = 5
readv(5, [{"", 0}, {"[global]\nlogfile=\"/dev/null\"\nplugin=\"/usr/lib/ulogd/ulogd_inppkt_NFLOG.so\"\nplugin=\"/usr/lib/ulogd/ulogd_raw2packet_BASE.so\"\nplugin=\"/usr/lib/ulogd/ulogd_output_PCAP.so\"\nstack=log1:NFLOG,base1:BASE,pcap1:PCAP\n\n[log1]\ngroup=86\n\n[pcap1]\nfile=\"/var/log/ulogd.pcap\"\nsync=1\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 1024}], 2) = 268
lseek(5, -34, SEEK_CUR)                 = 234
close(5)                                = 0
brk(0x55d724d32000)                     = 0x55d724d32000
socket(PF_NETLINK, SOCK_RAW, NETLINK_NETFILTER) = 5
getsockname(5, {sa_family=AF_NETLINK, pid=0, groups=00000000}, [12]) = 0
clock_gettime(CLOCK_REALTIME, {1479417537, 498032632}) = 0
bind(5, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
getsockname(5, {sa_family=AF_NETLINK, pid=25878, groups=00000000}, [12]) = 0
bind(5, {sa_family=AF_NETLINK, pid=25878, groups=00000000}, 12) = 0
sendto(5, "\34\0\0\0\1\4\5\0\0\0\0\0\0\0\0\0\0\0\0V\5\0\1\0\1\0\0\0", 28, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 28
recvfrom(5, "0\0\0\0\2\0\0\0\0\0\0\0\26e\0\0\377\377\377\377\34\0\0\0\1\4\5\0\0\0\0\0\0\0\0\0\0\0\0V\5\0\1\0\1\0\0\0", 8192, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, [12]) = 48
clock_gettime(CLOCK_REALTIME, {1479417537, 498577692}) = 0
writev(2, [{"Thu Nov 17 21:18:57 2016 <7> ulogd_inppkt_NFLOG.c:552 ", 54}, {NULL, 0}], 2Thu Nov 17 21:18:57 2016 <7> ulogd_inppkt_NFLOG.c:552 ) = 54
writev(4, [{"Thu Nov 17 21:18:57 2016 <7> ulogd_inppkt_NFLOG.c:552 unable to bind to log group 86\n", 85}, {NULL, 0}], 2) = 85
writev(2, [{"unable to bind to log group 86\n", 31}, {NULL, 0}], 2unable to bind to log group 86
) = 31
close(5)                                = 0
clock_gettime(CLOCK_REALTIME, {1479417537, 499088549}) = 0
writev(2, [{"Thu Nov 17 21:18:57 2016 <7> ulogd.c:813 ", 41}, {NULL, 0}], 2Thu Nov 17 21:18:57 2016 <7> ulogd.c:813 ) = 41
writev(4, [{"Thu Nov 17 21:18:57 2016 <7> ulogd.c:813 error starting `log1'\n", 63}, {NULL, 0}], 2) = 63
writev(2, [{"error starting `log1'\n", 22}, {NULL, 0}], 2error starting `log1'
) = 22
lseek(3, -52, SEEK_CUR)                 = 216
close(3)                                = 0
clock_gettime(CLOCK_REALTIME, {1479417537, 499672832}) = 0
writev(2, [{"Thu Nov 17 21:18:57 2016 <8> ulogd.c:1430 ", 42}, {NULL, 0}], 2Thu Nov 17 21:18:57 2016 <8> ulogd.c:1430 ) = 42
writev(4, [{"Thu Nov 17 21:18:57 2016 <8> ulogd.c:1430 not even a single working plugin stack\n", 81}, {NULL, 0}], 2) = 81
writev(2, [{"not even a single working plugin stack\n", 39}, {NULL, 0}], 2not even a single working plugin stack
) = 39
writev(2, [{"", 0}, {"Fatal error.\n", 13}], 2Fatal error.
) = 13
exit_group(1)                           = ?
+++ exited with 1 +++
/ # ulogd -v
Thu Nov 17 21:21:30 2016 <5> ulogd.c:843 building new pluginstance stack: 'log1:NFLOG,base1:BASE,pcap1:PCAP'
Thu Nov 17 21:21:30 2016 <7> ulogd_inppkt_NFLOG.c:552 unable to bind to log group 86
Thu Nov 17 21:21:30 2016 <7> ulogd.c:813 error starting `log1'
Thu Nov 17 21:21:30 2016 <8> ulogd.c:1430 not even a single working plugin stack
Fatal error.```
@chrislovecnm chrislovecnm changed the title kube-weave weave-npc error kube-weave weave-npc failing to start Nov 17, 2016
@chrislovecnm
Copy link
Author

https://github.com/inliniac/ulogd2/blob/master/input/packet/ulogd_inppkt_NFLOG.c#L550 is where it is dieing

@awh
Copy link
Contributor

awh commented Nov 18, 2016

Thanks for the report @chrislovecnm. Very odd that it fails on only one of the machines - I suspect you would have mentioned it already if it were the case, but is there anything different about that one?

@awh awh self-assigned this Nov 18, 2016
@awh
Copy link
Contributor

awh commented Nov 18, 2016

@chrislovecnm two possibilities come to mind so far, most likely first:

  1. Something is already bound to that group - most likely another instance of ulogd - observe:
vagrant@vagrant-ubuntu-wily-64:~$ sudo docker run -tid --entrypoint=ulogd --privileged --net=host weaveworks/weave-npc:1.8.0 -v
87f7c2a86d7fb275b74c2ed5e1779b7453dd3234e47ef50b921ed0eabfef9f3f
vagrant@vagrant-ubuntu-wily-64:~$ docker logs 87f7c2a86d7fb275b74c2ed5e1779b7453dd3234e47ef50b921ed0eabfef9f3f
Fri Nov 18 12:32:31 2016 <5> ulogd.c:843 building new pluginstance stack: 'log1:NFLOG,base1:BASE,pcap1:PCAP'
vagrant@vagrant-ubuntu-wily-64:~$ sudo docker run -ti --entrypoint=ulogd --privileged --net=host weaveworks/weave-npc:1.8.0 -v
Fri Nov 18 12:32:50 2016 <5> ulogd.c:843 building new pluginstance stack: 'log1:NFLOG,base1:BASE,pcap1:PCAP'
Fri Nov 18 12:32:50 2016 <7> ulogd_inppkt_NFLOG.c:552 unable to bind to log group 86
Fri Nov 18 12:32:50 2016 <7> ulogd.c:813 error starting `log1'
Fri Nov 18 12:32:50 2016 <8> ulogd.c:1430 not even a single working plugin stack
Fatal error.

I wonder if perhaps we have somehow ended up with an orphaned ulogd process lying around on that machine?

  1. The netlink bind attempts to load some kernel modules - do you have nfnetlink and nfnetlink_log modules available on that machine?
vagrant@vagrant-ubuntu-wily-64:~$ lsmod|grep nfnetlink
vagrant@vagrant-ubuntu-wily-64:~$ find /lib/modules|grep nfnetlink
/lib/modules/4.2.0-27-generic/kernel/net/netfilter/nfnetlink_queue.ko
/lib/modules/4.2.0-27-generic/kernel/net/netfilter/nfnetlink_log.ko
/lib/modules/4.2.0-27-generic/kernel/net/netfilter/nfnetlink_acct.ko
/lib/modules/4.2.0-27-generic/kernel/net/netfilter/nfnetlink.ko
/lib/modules/4.2.0-27-generic/kernel/net/netfilter/nfnetlink_cttimeout.ko
/lib/modules/4.2.0-27-generic/kernel/net/netfilter/nfnetlink_cthelper.ko
vagrant@vagrant-ubuntu-wily-64:~$ sudo mv /lib/modules/4.2.0-27-generic/kernel/net/netfilter/nfnetlink_log.ko /tmp
vagrant@vagrant-ubuntu-wily-64:~$ sudo docker run -ti --entrypoint=ulogd --privileged --net=host weaveworks/weave-npc:1.8.0 -v
Fri Nov 18 12:18:37 2016 <5> ulogd.c:843 building new pluginstance stack: 'log1:NFLOG,base1:BASE,pcap1:PCAP'
Fri Nov 18 12:18:37 2016 <7> ulogd_inppkt_NFLOG.c:552 unable to bind to log group 86
Fri Nov 18 12:18:37 2016 <7> ulogd.c:813 error starting `log1'
Fri Nov 18 12:18:37 2016 <8> ulogd.c:1430 not even a single working plugin stack
Fatal error.

@chrislovecnm
Copy link
Author

All the machines are the same ami, let me poke around. We have not been able to recreate it ;(

@awh
Copy link
Contributor

awh commented Nov 18, 2016

Thanks. I'm going to try to see if I can get a ulogd to survive the weave-npc pod exiting - that would clash once k8s restarts the pod.

@chrislovecnm
Copy link
Author

Winner winner chicken dinner... ulogd is already running

admin@ip-172-20-155-50:~$ ps axu  | grep ulogd
root      1911  0.0  0.0  13528   624 ?        S<   Nov17   0:00 /usr/sbin/ulogd -v
admin    25256  0.0  0.0  12732  2224 pts/0    S+   15:18   0:00 grep ulogd

docker info

admin@ip-172-20-155-50:~$ sudo docker info
Containers: 17
 Running: 12
 Paused: 0
 Stopped: 5
Images: 7
Server Version: 1.11.2
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge
Kernel Version: 4.4.26-k8s
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.42 GiB
Name: ip-172-20-155-50
ID: YP6U:EMZ2:PGZU:7TFB:N7HC:EF66:R3GB:I66C:UWJQ:KFIH:YPW3:AIKV
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
WARNING: No kernel memory limit support

@chrislovecnm
Copy link
Author

So killed ulogd and weave-npc has started.

weave pod is now not happy

admin@ip-172-20-138-109:~$ kubectl -n kube-system logs weave-net-3uo2y weave
INFO: 2016/11/18 15:31:15.560614 Command line options: map[docker-api: datapath:datapath ipalloc-init:consensus=23 name:0e:f1:90:a6:be:75 nickname:ip-172-20-155-50 http-addr:127.0.0.1:6784 ipalloc-range:10.32.0.0/12 no-dns:true port:6783 status-addr:0.0.0.0:6782]
INFO: 2016/11/18 15:31:15.586633 Communication between peers is unencrypted.
INFO: 2016/11/18 15:31:15.587966 Our name is 0e:f1:90:a6:be:75(ip-172-20-155-50)
INFO: 2016/11/18 15:31:15.588011 Launch detected - using supplied peer list: [172.20.138.109 172.20.141.149 172.20.141.150 172.20.150.181 172.20.155.49 172.20.155.50 172.20.156.101 172.20.156.102 172.20.156.103 172.20.156.104 172.20.165.105 172.20.165.106 172.20.166.169 172.20.169.80 172.20.169.81 172.20.185.17 172.20.203.33 172.20.207.131 172.20.219.33 172.20.219.34 172.20.219.35 172.20.219.36 172.20.220.184]
INFO: 2016/11/18 15:31:15.588257 [allocator 0e:f1:90:a6:be:75] Initialising with persisted data
INFO: 2016/11/18 15:31:15.588286 Sniffing traffic on datapath (via ODP)
INFO: 2016/11/18 15:31:15.590026 ->[172.20.138.109:6783] attempting connection
INFO: 2016/11/18 15:31:15.590236 ->[172.20.169.80:6783] attempting connection
INFO: 2016/11/18 15:31:15.590330 ->[172.20.207.131:6783] attempting connection
INFO: 2016/11/18 15:31:15.590413 ->[172.20.219.36:6783] attempting connection
INFO: 2016/11/18 15:31:15.590492 ->[172.20.156.101:6783] attempting connection
INFO: 2016/11/18 15:31:15.590587 ->[172.20.203.33:6783] attempting connection
INFO: 2016/11/18 15:31:15.590664 ->[172.20.185.17:6783] attempting connection
INFO: 2016/11/18 15:31:15.590745 ->[172.20.156.102:6783] attempting connection
INFO: 2016/11/18 15:31:15.590842 ->[172.20.165.106:6783] attempting connection
INFO: 2016/11/18 15:31:15.590945 ->[172.20.219.35:6783] attempting connection
INFO: 2016/11/18 15:31:15.591037 ->[172.20.141.150:6783] attempting connection
INFO: 2016/11/18 15:31:15.591162 ->[172.20.155.50:6783] attempting connection
INFO: 2016/11/18 15:31:15.591460 ->[172.20.156.104:6783] attempting connection
INFO: 2016/11/18 15:31:15.591554 Discovered local MAC 0e:f1:90:a6:be:75
INFO: 2016/11/18 15:31:15.591564 ->[172.20.219.33:6783] attempting connection
INFO: 2016/11/18 15:31:15.591721 ->[172.20.219.34:6783] attempting connection
INFO: 2016/11/18 15:31:15.591817 ->[172.20.155.49:6783] attempting connection
INFO: 2016/11/18 15:31:15.591912 ->[172.20.169.81:6783] attempting connection
INFO: 2016/11/18 15:31:15.592062 ->[172.20.165.105:6783] attempting connection
INFO: 2016/11/18 15:31:15.592181 ->[172.20.156.103:6783] attempting connection
INFO: 2016/11/18 15:31:15.592299 ->[172.20.141.149:6783] attempting connection
INFO: 2016/11/18 15:31:15.592424 ->[172.20.220.184:6783] attempting connection
INFO: 2016/11/18 15:31:15.592554 ->[172.20.150.181:6783] attempting connection
INFO: 2016/11/18 15:31:15.592795 ->[172.20.166.169:6783] attempting connection
INFO: 2016/11/18 15:31:15.593124 ->[172.20.155.50:56092] connection accepted
INFO: 2016/11/18 15:31:15.594038 ->[172.20.155.49:6783|2e:6d:be:5d:27:61(ip-172-20-155-49)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.597373 ->[172.20.156.102:6783|ee:17:93:3e:63:c0(ip-172-20-156-102)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.598296 ->[172.20.141.150:6783|62:1e:21:4b:d9:88(ip-172-20-141-150)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.598396 ->[172.20.219.36:6783|0a:9a:ee:de:9a:88(ip-172-20-219-36)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.598503 ->[172.20.156.104:6783|be:f1:13:a7:81:54(ip-172-20-156-104)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.599688 ->[172.20.185.17:6783|3e:1e:3c:45:07:fc(ip-172-20-185-17)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.599789 ->[172.20.138.109:6783|42:3e:21:7c:f2:43(ip-172-20-138-109)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.599904 ->[172.20.141.149:6783|ee:48:33:45:23:15(ip-172-20-141-149)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.600044 ->[172.20.169.80:6783|3a:94:bd:27:ff:cb(ip-172-20-169-80)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.615557 overlay_switch ->[be:f1:13:a7:81:54(ip-172-20-156-104)] using fastdp
INFO: 2016/11/18 15:31:15.615649 overlay_switch ->[ee:17:93:3e:63:c0(ip-172-20-156-102)] using fastdp
INFO: 2016/11/18 15:31:15.615683 Listening for HTTP control messages on 127.0.0.1:6784
INFO: 2016/11/18 15:31:15.615699 overlay_switch ->[0a:9a:ee:de:9a:88(ip-172-20-219-36)] using fastdp
INFO: 2016/11/18 15:31:15.615593 overlay_switch ->[3a:94:bd:27:ff:cb(ip-172-20-169-80)] using fastdp
INFO: 2016/11/18 15:31:15.615703 overlay_switch ->[42:3e:21:7c:f2:43(ip-172-20-138-109)] using fastdp
INFO: 2016/11/18 15:31:15.615709 overlay_switch ->[ee:48:33:45:23:15(ip-172-20-141-149)] using fastdp
INFO: 2016/11/18 15:31:15.600156 ->[172.20.166.169:6783|b2:0e:6f:b5:46:d4(ip-172-20-166-169)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.615785 overlay_switch ->[3e:1e:3c:45:07:fc(ip-172-20-185-17)] using fastdp
INFO: 2016/11/18 15:31:15.615819 overlay_switch ->[b2:0e:6f:b5:46:d4(ip-172-20-166-169)] using fastdp
INFO: 2016/11/18 15:31:15.615570 overlay_switch ->[2e:6d:be:5d:27:61(ip-172-20-155-49)] using fastdp
INFO: 2016/11/18 15:31:15.615839 ->[172.20.155.50:56092|0e:f1:90:a6:be:75(ip-172-20-155-50)]: connection shutting down due to error: cannot connect to ourself
INFO: 2016/11/18 15:31:15.615885 ->[172.20.156.101:6783|7e:4d:6d:1a:98:81(ip-172-20-156-101)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.615740 overlay_switch ->[62:1e:21:4b:d9:88(ip-172-20-141-150)] using fastdp
INFO: 2016/11/18 15:31:15.615933 overlay_switch ->[7e:4d:6d:1a:98:81(ip-172-20-156-101)] using fastdp
INFO: 2016/11/18 15:31:15.615942 ->[172.20.203.33:6783|b6:4e:7e:30:c2:9f(ip-172-20-203-33)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.615974 ->[172.20.219.34:6783|6a:57:8a:0c:a1:bc(ip-172-20-219-34)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.615987 overlay_switch ->[b6:4e:7e:30:c2:9f(ip-172-20-203-33)] using fastdp
INFO: 2016/11/18 15:31:15.616011 ->[172.20.150.181:6783|d6:77:48:29:c7:7f(ip-172-20-150-181)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.616037 overlay_switch ->[6a:57:8a:0c:a1:bc(ip-172-20-219-34)] using fastdp
INFO: 2016/11/18 15:31:15.616059 overlay_switch ->[d6:77:48:29:c7:7f(ip-172-20-150-181)] using fastdp
INFO: 2016/11/18 15:31:15.616086 ->[172.20.207.131:40254] connection accepted
INFO: 2016/11/18 15:31:15.616145 ->[172.20.207.131:6783|76:79:c2:0a:a4:f6(ip-172-20-207-131)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.616155 Listening for metrics requests on 0.0.0.0:6782
INFO: 2016/11/18 15:31:15.616252 ->[172.20.169.81:6783|02:28:20:6a:6f:37(ip-172-20-169-81)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.616265 overlay_switch ->[76:79:c2:0a:a4:f6(ip-172-20-207-131)] using fastdp
INFO: 2016/11/18 15:31:15.616310 overlay_switch ->[02:28:20:6a:6f:37(ip-172-20-169-81)] using fastdp
INFO: 2016/11/18 15:31:15.616315 ->[172.20.155.50:6783|0e:f1:90:a6:be:75(ip-172-20-155-50)]: connection shutting down due to error: cannot connect to ourself
INFO: 2016/11/18 15:31:15.616363 ->[172.20.219.33:6783|ca:79:51:d4:44:bf(ip-172-20-219-33)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.616409 overlay_switch ->[ca:79:51:d4:44:bf(ip-172-20-219-33)] using fastdp
INFO: 2016/11/18 15:31:15.616425 ->[172.20.165.105:6783|a6:78:18:3b:d6:a8(ip-172-20-165-105)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.616481 overlay_switch ->[a6:78:18:3b:d6:a8(ip-172-20-165-105)] using fastdp
INFO: 2016/11/18 15:31:15.616525 ->[172.20.219.35:6783|92:2b:b5:5f:cd:bc(ip-172-20-219-35)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.616588 overlay_switch ->[92:2b:b5:5f:cd:bc(ip-172-20-219-35)] using fastdp
INFO: 2016/11/18 15:31:15.616628 ->[172.20.165.106:6783|3e:b7:e2:91:4b:c7(ip-172-20-165-106)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.616675 ->[172.20.156.103:6783|be:78:0b:0c:8c:10(ip-172-20-156-103)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.616723 overlay_switch ->[be:78:0b:0c:8c:10(ip-172-20-156-103)] using fastdp
INFO: 2016/11/18 15:31:15.616739 ->[172.20.220.184:6783|1a:bf:07:42:11:b0(ip-172-20-220-184)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.616770 overlay_switch ->[1a:bf:07:42:11:b0(ip-172-20-220-184)] using fastdp
INFO: 2016/11/18 15:31:15.616807 ->[172.20.156.104:6783|be:f1:13:a7:81:54(ip-172-20-156-104)]: connection added (new peer)
INFO: 2016/11/18 15:31:15.616956 ->[172.20.156.102:6783|ee:17:93:3e:63:c0(ip-172-20-156-102)]: connection added (new peer)
INFO: 2016/11/18 15:31:15.617015 ->[172.20.165.105:43611] connection accepted
INFO: 2016/11/18 15:31:15.617071 ->[172.20.165.106:51517] connection accepted
INFO: 2016/11/18 15:31:15.617112 ->[172.20.220.184:40901] connection accepted
INFO: 2016/11/18 15:31:15.617250 ->[172.20.207.131:40254|76:79:c2:0a:a4:f6(ip-172-20-207-131)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.617280 ->[172.20.219.36:6783|0a:9a:ee:de:9a:88(ip-172-20-219-36)]: connection added (new peer)
INFO: 2016/11/18 15:31:15.617314 overlay_switch ->[76:79:c2:0a:a4:f6(ip-172-20-207-131)] using fastdp
INFO: 2016/11/18 15:31:15.617320 overlay_switch ->[3e:b7:e2:91:4b:c7(ip-172-20-165-106)] using fastdp
INFO: 2016/11/18 15:31:15.617384 ->[172.20.169.80:6783|3a:94:bd:27:ff:cb(ip-172-20-169-80)]: connection added (new peer)
INFO: 2016/11/18 15:31:15.617479 ->[172.20.138.109:6783|42:3e:21:7c:f2:43(ip-172-20-138-109)]: connection added (new peer)
INFO: 2016/11/18 15:31:15.617889 ->[172.20.141.149:6783|ee:48:33:45:23:15(ip-172-20-141-149)]: connection added (new peer)
INFO: 2016/11/18 15:31:15.619654 ->[172.20.165.105:43611|a6:78:18:3b:d6:a8(ip-172-20-165-105)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.619793 overlay_switch ->[a6:78:18:3b:d6:a8(ip-172-20-165-105)] using fastdp
INFO: 2016/11/18 15:31:15.622716 ->[172.20.165.106:51517|3e:b7:e2:91:4b:c7(ip-172-20-165-106)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.622793 overlay_switch ->[3e:b7:e2:91:4b:c7(ip-172-20-165-106)] using fastdp
INFO: 2016/11/18 15:31:15.624339 ->[172.20.220.184:40901|1a:bf:07:42:11:b0(ip-172-20-220-184)]: connection ready; using protocol version 2
INFO: 2016/11/18 15:31:15.624387 overlay_switch ->[1a:bf:07:42:11:b0(ip-172-20-220-184)] using fastdp
INFO: 2016/11/18 15:31:15.624774 ->[172.20.185.17:6783|3e:1e:3c:45:07:fc(ip-172-20-185-17)]: connection added (new peer)
INFO: 2016/11/18 15:31:15.625559 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2016/11/18 15:31:15.626694 ->[172.20.166.169:6783|b2:0e:6f:b5:46:d4(ip-172-20-166-169)]: connection added (new peer)
INFO: 2016/11/18 15:31:15.627748 ->[172.20.155.49:6783|2e:6d:be:5d:27:61(ip-172-20-155-49)]: connection added (new peer)
INFO: 2016/11/18 15:31:15.630314 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2016/11/18 15:31:15.632820 sleeve ->[172.20.169.80:6783|3a:94:bd:27:ff:cb(ip-172-20-169-80)]: Effective MTU verified at 8939
INFO: 2016/11/18 15:31:15.632936 ->[172.20.141.150:6783|62:1e:21:4b:d9:88(ip-172-20-141-150)]: connection added
INFO: 2016/11/18 15:31:15.632966 sleeve ->[172.20.219.36:6783|0a:9a:ee:de:9a:88(ip-172-20-219-36)]: Effective MTU verified at 8939
INFO: 2016/11/18 15:31:15.633172 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2016/11/18 15:31:15.633571 ->[172.20.156.101:6783|7e:4d:6d:1a:98:81(ip-172-20-156-101)]: connection added
INFO: 2016/11/18 15:31:15.634085 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2016/11/18 15:31:15.634606 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2016/11/18 15:31:15.634931 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2016/11/18 15:31:15.635184 ->[172.20.203.33:6783|b6:4e:7e:30:c2:9f(ip-172-20-203-33)]: connection added
INFO: 2016/11/18 15:31:15.635562 sleeve ->[172.20.141.149:6783|ee:48:33:45:23:15(ip-172-20-141-149)]: Effective MTU verified at 8939
INFO: 2016/11/18 15:31:15.637933 ->[172.20.219.34:6783|6a:57:8a:0c:a1:bc(ip-172-20-219-34)]: connection added
INFO: 2016/11/18 15:31:15.638103 sleeve ->[172.20.156.104:6783|be:f1:13:a7:81:54(ip-172-20-156-104)]: Effective MTU verified at 8939
INFO: 2016/11/18 15:31:15.638475 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2016/11/18 15:31:15.639826 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2016/11/18 15:31:15.639892 overlay_switch ->[3e:1e:3c:45:07:fc(ip-172-20-185-17)] using sleeve
INFO: 2016/11/18 15:31:15.640082 overlay_switch ->[3e:1e:3c:45:07:fc(ip-172-20-185-17)] using fastdp
INFO: 2016/11/18 15:31:15.641361 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2016/11/18 15:31:15.641560 ->[172.20.150.181:6783|d6:77:48:29:c7:7f(ip-172-20-150-181)]: connection added
INFO: 2016/11/18 15:31:15.642186 sleeve ->[172.20.138.109:6783|42:3e:21:7c:f2:43(ip-172-20-138-109)]: Effective MTU verified at 8939
INFO: 2016/11/18 15:31:15.643051 ->[172.20.207.131:6783|76:79:c2:0a:a4:f6(ip-172-20-207-131)]: connection added
INFO: 2016/11/18 15:31:15.643216 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2016/11/18 15:31:15.643674 sleeve ->[172.20.166.169:6783|b2:0e:6f:b5:46:d4(ip-172-20-166-169)]: Effective MTU verified at 8939
INFO: 2016/11/18 15:31:15.643804 sleeve ->[172.20.156.102:6783|ee:17:93:3e:63:c0(ip-172-20-156-102)]: Effective MTU verified at 8939
INFO: 2016/11/18 15:31:15.646049 sleeve ->[172.20.185.17:6783|3e:1e:3c:45:07:fc(ip-172-20-185-17)]: Effective MTU verified at 8939
INFO: 2016/11/18 15:31:15.646680 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2016/11/18 15:31:15.646934 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2016/11/18 15:31:15.647567 ->[172.20.169.81:6783|02:28:20:6a:6f:37(ip-172-20-169-81)]: connection added
INFO: 2016/11/18 15:31:15.647725 ->[172.20.219.33:6783|ca:79:51:d4:44:bf(ip-172-20-219-33)]: connection added
INFO: 2016/11/18 15:31:15.647982 sleeve ->[172.20.219.34:6783|6a:57:8a:0c:a1:bc(ip-172-20-219-34)]: Effective MTU verified at 8939
INFO: 2016/11/18 15:31:15.648150 ->[172.20.219.35:6783|92:2b:b5:5f:cd:bc(ip-172-20-219-35)]: connection added
INFO: 2016/11/18 15:31:15.648840 sleeve ->[172.20.155.49:6783|2e:6d:be:5d:27:61(ip-172-20-155-49)]: Effective MTU verified at 8939
INFO: 2016/11/18 15:31:15.652666 sleeve ->[172.20.156.101:6783|7e:4d:6d:1a:98:81(ip-172-20-156-101)]: Effective MTU verified at 8939
INFO: 2016/11/18 15:31:15.652726 ->[172.20.165.105:6783|a6:78:18:3b:d6:a8(ip-172-20-165-105)]: connection added
INFO: 2016/11/18 15:31:15.653549 ->[172.20.156.103:6783|be:78:0b:0c:8c:10(ip-172-20-156-103)]: connection added
INFO: 2016/11/18 15:31:15.653723 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2016/11/18 15:31:15.653848 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2016/11/18 15:31:15.654280 sleeve ->[172.20.203.33:6783|b6:4e:7e:30:c2:9f(ip-172-20-203-33)]: Effective MTU verified at 8939
INFO: 2016/11/18 15:31:15.654498 ->[172.20.165.105:6783|a6:78:18:3b:d6:a8(ip-172-20-165-105)]: connection shutting down due to error: write tcp4 172.20.155.50:37177->172.20.165.105:6783: write: connection reset by peer
INFO: 2016/11/18 15:31:15.655965 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2016/11/18 15:31:15.656765 ->[172.20.220.184:6783|1a:bf:07:42:11:b0(ip-172-20-220-184)]: connection added
INFO: 2016/11/18 15:31:15.657638 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2016/11/18 15:31:15.657725 sleeve ->[172.20.207.131:6783|76:79:c2:0a:a4:f6(ip-172-20-207-131)]: Effective MTU verified at 8939
INFO: 2016/11/18 15:31:15.659348 sleeve ->[172.20.141.150:6783|62:1e:21:4b:d9:88(ip-172-20-141-150)]: Effective MTU verified at 8939
INFO: 2016/11/18 15:31:15.659942 sleeve ->[172.20.219.33:6783|ca:79:51:d4:44:bf(ip-172-20-219-33)]: Effective MTU verified at 8939
INFO: 2016/11/18 15:31:15.660055 ->[172.20.165.106:6783|3e:b7:e2:91:4b:c7(ip-172-20-165-106)]: connection added
INFO: 2016/11/18 15:31:15.660133 ->[172.20.207.131:40254|76:79:c2:0a:a4:f6(ip-172-20-207-131)]: connection shutting down due to error: Multiple connections to 76:79:c2:0a:a4:f6(ip-172-20-207-131) added to 0e:f1:90:a6:be:75(ip-172-20-155-50)
INFO: 2016/11/18 15:31:15.660086 sleeve ->[172.20.150.181:6783|d6:77:48:29:c7:7f(ip-172-20-150-181)]: Effective MTU verified at 8939
INFO: 2016/11/18 15:31:15.660373 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2016/11/18 15:31:15.660898 ->[172.20.165.106:6783|3e:b7:e2:91:4b:c7(ip-172-20-165-106)]: connection deleted
INFO: 2016/11/18 15:31:15.660959 ->[172.20.165.106:6783|3e:b7:e2:91:4b:c7(ip-172-20-165-106)]: connection shutting down due to error: Multiple connections to 3e:b7:e2:91:4b:c7(ip-172-20-165-106) added to 0e:f1:90:a6:be:75(ip-172-20-155-50)
INFO: 2016/11/18 15:31:15.661392 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2016/11/18 15:31:15.661474 overlay_switch ->[be:78:0b:0c:8c:10(ip-172-20-156-103)] using sleeve
INFO: 2016/11/18 15:31:15.661579 overlay_switch ->[be:78:0b:0c:8c:10(ip-172-20-156-103)] using fastdp
INFO: 2016/11/18 15:31:15.661793 sleeve ->[172.20.219.35:6783|92:2b:b5:5f:cd:bc(ip-172-20-219-35)]: Effective MTU verified at 8939
INFO: 2016/11/18 15:31:15.662194 sleeve ->[172.20.156.103:6783|be:78:0b:0c:8c:10(ip-172-20-156-103)]: Effective MTU verified at 8939
INFO: 2016/11/18 15:31:15.663666 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2016/11/18 15:31:15.664001 ->[172.20.165.106:51517|3e:b7:e2:91:4b:c7(ip-172-20-165-106)]: connection added
INFO: 2016/11/18 15:31:15.664281 ->[172.20.156.104:6783|be:f1:13:a7:81:54(ip-172-20-156-104)]: connection fully established
INFO: 2016/11/18 15:31:15.664758 ->[172.20.219.36:6783|0a:9a:ee:de:9a:88(ip-172-20-219-36)]: connection fully established
INFO: 2016/11/18 15:31:15.664891 ->[172.20.220.184:40901|1a:bf:07:42:11:b0(ip-172-20-220-184)]: connection shutting down due to error: Multiple connections to 1a:bf:07:42:11:b0(ip-172-20-220-184) added to 0e:f1:90:a6:be:75(ip-172-20-155-50)
INFO: 2016/11/18 15:31:15.665444 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2016/11/18 15:31:15.665591 overlay_switch ->[02:28:20:6a:6f:37(ip-172-20-169-81)] using sleeve
INFO: 2016/11/18 15:31:15.665628 overlay_switch ->[02:28:20:6a:6f:37(ip-172-20-169-81)] using fastdp
INFO: 2016/11/18 15:31:15.666575 ->[172.20.169.80:6783|3a:94:bd:27:ff:cb(ip-172-20-169-80)]: connection fully established
INFO: 2016/11/18 15:31:15.667359 ->[172.20.165.105:6783|a6:78:18:3b:d6:a8(ip-172-20-165-105)]: connection deleted
INFO: 2016/11/18 15:31:15.668603 sleeve ->[172.20.220.184:6783|1a:bf:07:42:11:b0(ip-172-20-220-184)]: Effective MTU verified at 8939
INFO: 2016/11/18 15:31:15.668684 sleeve ->[172.20.169.81:6783|02:28:20:6a:6f:37(ip-172-20-169-81)]: Effective MTU verified at 8939
INFO: 2016/11/18 15:31:15.670251 ->[172.20.165.105:43611|a6:78:18:3b:d6:a8(ip-172-20-165-105)]: connection added
INFO: 2016/11/18 15:31:15.671164 ->[172.20.156.101:6783|7e:4d:6d:1a:98:81(ip-172-20-156-101)]: connection fully established
INFO: 2016/11/18 15:31:15.672770 ->[172.20.185.17:6783|3e:1e:3c:45:07:fc(ip-172-20-185-17)]: connection fully established
INFO: 2016/11/18 15:31:15.677491 ->[172.20.156.102:6783|ee:17:93:3e:63:c0(ip-172-20-156-102)]: connection fully established
INFO: 2016/11/18 15:31:15.679480 ->[172.20.166.169:6783|b2:0e:6f:b5:46:d4(ip-172-20-166-169)]: connection fully established
INFO: 2016/11/18 15:31:15.680656 ->[172.20.141.149:6783|ee:48:33:45:23:15(ip-172-20-141-149)]: connection fully established
INFO: 2016/11/18 15:31:15.682159 ->[172.20.155.49:6783|2e:6d:be:5d:27:61(ip-172-20-155-49)]: connection fully established
INFO: 2016/11/18 15:31:15.683390 ->[172.20.138.109:6783|42:3e:21:7c:f2:43(ip-172-20-138-109)]: connection fully established
INFO: 2016/11/18 15:31:15.684370 ->[172.20.203.33:6783|b6:4e:7e:30:c2:9f(ip-172-20-203-33)]: connection fully established
INFO: 2016/11/18 15:31:15.686034 ->[172.20.219.35:6783|92:2b:b5:5f:cd:bc(ip-172-20-219-35)]: connection fully established
INFO: 2016/11/18 15:31:15.687016 ->[172.20.219.34:6783|6a:57:8a:0c:a1:bc(ip-172-20-219-34)]: connection fully established
INFO: 2016/11/18 15:31:15.688039 ->[172.20.150.181:6783|d6:77:48:29:c7:7f(ip-172-20-150-181)]: connection fully established
INFO: 2016/11/18 15:31:15.688788 ->[172.20.219.33:6783|ca:79:51:d4:44:bf(ip-172-20-219-33)]: connection fully established
INFO: 2016/11/18 15:31:15.701568 ->[172.20.141.150:6783|62:1e:21:4b:d9:88(ip-172-20-141-150)]: connection fully established
INFO: 2016/11/18 15:31:15.702162 ->[172.20.156.103:6783|be:78:0b:0c:8c:10(ip-172-20-156-103)]: connection fully established
INFO: 2016/11/18 15:31:15.702652 ->[172.20.220.184:6783|1a:bf:07:42:11:b0(ip-172-20-220-184)]: connection fully established
INFO: 2016/11/18 15:31:15.703245 ->[172.20.207.131:6783|76:79:c2:0a:a4:f6(ip-172-20-207-131)]: connection fully established
INFO: 2016/11/18 15:31:15.704045 ->[172.20.169.81:6783|02:28:20:6a:6f:37(ip-172-20-169-81)]: connection fully established
INFO: 2016/11/18 15:31:15.743992 Discovered remote MAC 6a:57:8a:0c:a1:bc at 6a:57:8a:0c:a1:bc(ip-172-20-219-34)
INFO: 2016/11/18 15:31:15.744205 Discovered local MAC 06:bc:80:22:a5:6e
INFO: 2016/11/18 15:31:15.819403 Discovered remote MAC 3a:94:bd:27:ff:cb at 3a:94:bd:27:ff:cb(ip-172-20-169-80)
INFO: 2016/11/18 15:31:15.819548 Discovered remote MAC b2:0e:6f:b5:46:d4 at b2:0e:6f:b5:46:d4(ip-172-20-166-169)
INFO: 2016/11/18 15:31:15.819680 Discovered remote MAC 02:28:20:6a:6f:37 at 02:28:20:6a:6f:37(ip-172-20-169-81)
INFO: 2016/11/18 15:31:15.819811 Discovered remote MAC a6:78:18:3b:d6:a8 at a6:78:18:3b:d6:a8(ip-172-20-165-105)
INFO: 2016/11/18 15:31:15.819958 Discovered remote MAC 3e:b7:e2:91:4b:c7 at 3e:b7:e2:91:4b:c7(ip-172-20-165-106)
INFO: 2016/11/18 15:31:15.820603 Discovered remote MAC 3e:1e:3c:45:07:fc at 3e:1e:3c:45:07:fc(ip-172-20-185-17)
INFO: 2016/11/18 15:31:16.121097 ->[172.20.165.106:51517|3e:b7:e2:91:4b:c7(ip-172-20-165-106)]: connection fully established
INFO: 2016/11/18 15:31:16.121263 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2016/11/18 15:31:16.122000 ->[172.20.165.105:43611|a6:78:18:3b:d6:a8(ip-172-20-165-105)]: connection fully established
INFO: 2016/11/18 15:31:16.122070 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2016/11/18 15:31:16.123586 sleeve ->[172.20.165.105:6783|a6:78:18:3b:d6:a8(ip-172-20-165-105)]: Effective MTU verified at 8939
INFO: 2016/11/18 15:31:16.123622 sleeve ->[172.20.165.106:6783|3e:b7:e2:91:4b:c7(ip-172-20-165-106)]: Effective MTU verified at 8939
iptables: No chain/target/match by that name.

@awh
Copy link
Contributor

awh commented Nov 18, 2016

iptables: No chain/target/match by that name.

See #2617, which will be fixed in 1.8.1.

@chrislovecnm chrislovecnm mentioned this issue Nov 18, 2016
Closed
@chrislovecnm
Copy link
Author

Filed #2654 to help with the pid0 stuff that we MAY have ran into

@awh awh added this to the 1.8.1 milestone Nov 18, 2016
@bboreham bboreham modified the milestones: 1.8.2, 1.8.1 Nov 21, 2016
@awh
Copy link
Contributor

awh commented Nov 29, 2016

Further analysis: weave-kube runs with hostPID: true, in which case docker should clean up any leftover containers in the container cgroup when the main process exits unexpectedly (see https://github.com/opencontainers/runc/blob/f59ba3cdd76fdc08c004f42aa915996f6f420899/libcontainer/state_linux.go#L41-L45).

@brb brb modified the milestones: 1.8.2, 1.8.3 Dec 8, 2016
@awh awh modified the milestones: 1.9.0, 1.8.3 Jan 11, 2017
@awh awh mentioned this issue Jan 11, 2017
Merged
bboreham added a commit that referenced this issue Jan 11, 2017
@bboreham
Merge pull request #2728 from /issues/2653-ulogd-pdeathsig
0145282 
SIGKILL ulogd on weave-npc exit
@brb brb mentioned this issue Jan 16, 2017
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@awh awh

Labels
bug
Projects
None yet
Milestone
1.9.0
Development

No branches or pull requests
4 participants
@brb @chrislovecnm @awh @bboreham