Crash while stopping via Sigterm (v 1.11)

[bemeyert]

Hello,

I just stumbled over a problem while running carbon-c-relay under CentOS 7 in version 1.11. When stopping the service with systemctl stop carbon-c-relay it crashes. This happens in a test-setup in a Vagrantbox.
Here the output from /var/log/messages:

[root@relay1 ~]# grep carbon /var/log/messages
Jun 14 10:06:06 localhost yum[9342]: Installed: carbon-c-relay-1.11-2.el7.x86_64
Jun 14 10:06:08 localhost carbon-c-relay: [2016-06-14 10:06:08] starting carbon-c-relay v1.11 (2016-05-18), pid=9397
Jun 14 10:06:08 localhost carbon-c-relay: configuration:
Jun 14 10:06:08 localhost carbon-c-relay: relay hostname = relay1.relay
Jun 14 10:06:08 localhost carbon-c-relay: listen port = 2003
Jun 14 10:06:08 localhost carbon-c-relay: workers = 1
Jun 14 10:06:08 localhost carbon-c-relay: send batch size = 2500
Jun 14 10:06:08 localhost carbon-c-relay: server queue size = 25000
Jun 14 10:06:08 localhost carbon-c-relay: statistics submission interval = 60s
Jun 14 10:06:08 localhost carbon-c-relay: listen backlog = 3
Jun 14 10:06:08 localhost carbon-c-relay: server connection IO timeout = 600ms
Jun 14 10:06:08 localhost carbon-c-relay: routes configuration = /etc/carbon-c-relay.conf
Jun 14 10:06:08 localhost carbon-c-relay: parsed configuration follows:
Jun 14 10:06:08 localhost carbon-c-relay: cluster replica_1
Jun 14 10:06:08 localhost carbon-c-relay: fnv1a_ch replication 1
Jun 14 10:06:08 localhost carbon-c-relay: cc1.relay:2001=f568b3afdc29ce0a932b191c0e980ece
Jun 14 10:06:08 localhost carbon-c-relay: cc1.relay:2002=d8d6781940eb8c057d6d6d5c888d7621
Jun 14 10:06:08 localhost carbon-c-relay: ;
Jun 14 10:06:08 localhost carbon-c-relay: cluster replica_2
Jun 14 10:06:08 localhost carbon-c-relay: fnv1a_ch replication 1
Jun 14 10:06:08 localhost carbon-c-relay: cc2.relay:2001=f568b3afdc29ce0a932b191c0e980ece
Jun 14 10:06:08 localhost carbon-c-relay: cc2.relay:2002=d8d6781940eb8c057d6d6d5c888d7621
Jun 14 10:06:08 localhost carbon-c-relay: ;
Jun 14 10:06:08 localhost carbon-c-relay: match *
Jun 14 10:06:08 localhost carbon-c-relay: send to
Jun 14 10:06:08 localhost carbon-c-relay: replica_1
Jun 14 10:06:08 localhost carbon-c-relay: replica_2
Jun 14 10:06:08 localhost carbon-c-relay: stop
Jun 14 10:06:08 localhost carbon-c-relay: ;
Jun 14 10:06:08 localhost carbon-c-relay: [2016-06-14 10:06:08] listening on tcp4 0.0.0.0 port 2003
Jun 14 10:06:08 localhost carbon-c-relay: [2016-06-14 10:06:08] listening on tcp6 :: port 2003
Jun 14 10:06:08 localhost carbon-c-relay: [2016-06-14 10:06:08] listening on udp4 0.0.0.0 port 2003
Jun 14 10:06:08 localhost carbon-c-relay: [2016-06-14 10:06:08] listening on udp6 :: port 2003
Jun 14 10:06:08 localhost carbon-c-relay: [2016-06-14 10:06:08] listening on UNIX socket /tmp/.s.carbon-c-relay.2003
Jun 14 10:06:08 localhost carbon-c-relay: [2016-06-14 10:06:08] starting 1 workers
Jun 14 10:06:08 localhost carbon-c-relay: [2016-06-14 10:06:08] starting statistics collector
Jun 14 10:06:08 localhost carbon-c-relay: [2016-06-14 10:06:08] startup sequence complete
Jun 14 10:19:47 localhost carbon-c-relay: [2016-06-14 10:19:47] caught SIGTERM
Jun 14 10:19:47 localhost carbon-c-relay: [2016-06-14 10:19:47] shutting down...
Jun 14 10:19:47 localhost carbon-c-relay: [2016-06-14 10:19:47] closed listeners for port 2003
Jun 14 10:19:48 localhost carbon-c-relay: [2016-06-14 10:19:48] stopped collector
Jun 14 10:19:48 localhost carbon-c-relay: [2016-06-14 10:19:48] stopped worker 1 2
Jun 14 10:19:48 localhost carbon-c-relay: *** Error in `/usr/bin/carbon-c-relay': free(): invalid pointer: 0x00007f551e2ad0a0 ***
Jun 14 10:19:48 localhost carbon-c-relay: ======= Backtrace: =========
Jun 14 10:19:48 localhost carbon-c-relay: /lib64/libc.so.6(+0x7d053)[0x7f551ccef053]
Jun 14 10:19:48 localhost carbon-c-relay: /usr/bin/carbon-c-relay(+0xf9ea)[0x7f551d7859ea]
Jun 14 10:19:48 localhost carbon-c-relay: /usr/bin/carbon-c-relay(+0x7604)[0x7f551d77d604]
Jun 14 10:19:48 localhost carbon-c-relay: /usr/bin/carbon-c-relay(+0xe090)[0x7f551d784090]
Jun 14 10:19:48 localhost carbon-c-relay: /usr/bin/carbon-c-relay(+0x3139)[0x7f551d779139]
Jun 14 10:19:48 localhost carbon-c-relay: /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f551cc93b15]
Jun 14 10:19:48 localhost carbon-c-relay: /usr/bin/carbon-c-relay(+0x36a9)[0x7f551d7796a9]
Jun 14 10:19:48 localhost carbon-c-relay: ======= Memory map: ========
Jun 14 10:19:48 localhost carbon-c-relay: 7f5508000000-7f5508021000 rw-p 00000000 00:00 0
Jun 14 10:19:48 localhost carbon-c-relay: 7f5508021000-7f550c000000 ---p 00000000 00:00 0
Jun 14 10:19:48 localhost carbon-c-relay: 7f5510000000-7f5510022000 rw-p 00000000 00:00 0
Jun 14 10:19:48 localhost carbon-c-relay: 7f5510022000-7f5514000000 ---p 00000000 00:00 0
Jun 14 10:19:48 localhost carbon-c-relay: 7f5517f2f000-7f5517f44000 r-xp 00000000 fd:00 33554573                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
Jun 14 10:19:48 localhost carbon-c-relay: 7f5517f44000-7f5518143000 ---p 00015000 fd:00 33554573                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
Jun 14 10:19:48 localhost carbon-c-relay: 7f5518143000-7f5518144000 r--p 00014000 fd:00 33554573                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
Jun 14 10:19:48 localhost carbon-c-relay: 7f5518144000-7f5518145000 rw-p 00015000 fd:00 33554573                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
Jun 14 10:19:48 localhost carbon-c-relay: 7f5518145000-7f5518146000 ---p 00000000 00:00 0
Jun 14 10:19:48 localhost carbon-c-relay: 7f5518146000-7f5518946000 rw-p 00000000 00:00 0
Jun 14 10:19:48 localhost carbon-c-relay: 7f5518946000-7f5518947000 ---p 00000000 00:00 0
Jun 14 10:19:48 localhost carbon-c-relay: 7f5518947000-7f5519147000 rw-p 00000000 00:00 0
Jun 14 10:19:48 localhost carbon-c-relay: 7f5519147000-7f5519148000 ---p 00000000 00:00 0
Jun 14 10:19:48 localhost carbon-c-relay: 7f5519148000-7f551aa5b000 rw-p 00000000 00:00 0
Jun 14 10:19:48 localhost carbon-c-relay: 7f551aa5b000-7f551aa5c000 ---p 00000000 00:00 0
Jun 14 10:19:48 localhost carbon-c-relay: 7f551aa5c000-7f551b25c000 rw-p 00000000 00:00 0                          [stack:9402]
Jun 14 10:19:48 localhost carbon-c-relay: 7f551b25c000-7f551b25d000 ---p 00000000 00:00 0
Jun 14 10:19:48 localhost carbon-c-relay: 7f551b25d000-7f551ba5d000 rw-p 00000000 00:00 0                          [stack:9401]
Jun 14 10:19:48 localhost carbon-c-relay: 7f551ba5d000-7f551ba5e000 ---p 00000000 00:00 0
Jun 14 10:19:48 localhost carbon-c-relay: 7f551ba5e000-7f551c25e000 rw-p 00000000 00:00 0                          [stack:9400]
Jun 14 10:19:48 localhost carbon-c-relay: 7f551c25e000-7f551c25f000 ---p 00000000 00:00 0
Jun 14 10:19:48 localhost carbon-c-relay: 7f551c25f000-7f551ca5f000 rw-p 00000000 00:00 0
Jun 14 10:19:48 localhost carbon-c-relay: 7f551ca5f000-7f551ca6b000 r-xp 00000000 fd:00 33990819                   /usr/lib64/libnss_files-2.17.so
Jun 14 10:19:48 localhost carbon-c-relay: 7f551ca6b000-7f551cc6a000 ---p 0000c000 fd:00 33990819                   /usr/lib64/libnss_files-2.17.so
Jun 14 10:19:48 localhost carbon-c-relay: 7f551cc6a000-7f551cc6b000 r--p 0000b000 fd:00 33990819                   /usr/lib64/libnss_files-2.17.so
Jun 14 10:19:48 localhost carbon-c-relay: 7f551cc6b000-7f551cc6c000 rw-p 0000c000 fd:00 33990819                   /usr/lib64/libnss_files-2.17.so
Jun 14 10:19:48 localhost carbon-c-relay: 7f551cc6c000-7f551cc72000 rw-p 00000000 00:00 0
Jun 14 10:19:48 localhost carbon-c-relay: 7f551cc72000-7f551ce29000 r-xp 00000000 fd:00 33990801                   /usr/lib64/libc-2.17.so
Jun 14 10:19:48 localhost carbon-c-relay: 7f551ce29000-7f551d029000 ---p 001b7000 fd:00 33990801                   /usr/lib64/libc-2.17.so
Jun 14 10:19:48 localhost carbon-c-relay: 7f551d029000-7f551d02d000 r--p 001b7000 fd:00 33990801                   /usr/lib64/libc-2.17.so
Jun 14 10:19:48 localhost carbon-c-relay: 7f551d02d000-7f551d02f000 rw-p 001bb000 fd:00 33990801                   /usr/lib64/libc-2.17.so
Jun 14 10:19:48 localhost carbon-c-relay: 7f551d02f000-7f551d034000 rw-p 00000000 00:00 0
Jun 14 10:19:48 localhost carbon-c-relay: 7f551d034000-7f551d04a000 r-xp 00000000 fd:00 33990827                   /usr/lib64/libpthread-2.17.so
Jun 14 10:19:48 localhost carbon-c-relay: 7f551d04a000-7f551d24a000 ---p 00016000 fd:00 33990827                   /usr/lib64/libpthread-2.17.so
Jun 14 10:19:48 localhost carbon-c-relay: 7f551d24a000-7f551d24b000 r--p 00016000 fd:00 33990827                   /usr/lib64/libpthread-2.17.so
Jun 14 10:19:48 localhost carbon-c-relay: 7f551d24b000-7f551d24c000 rw-p 00017000 fd:00 33990827                   /usr/lib64/libpthread-2.17.so
Jun 14 10:19:48 localhost carbon-c-relay: 7f551d24c000-7f551d250000 rw-p 00000000 00:00 0
Jun 14 10:19:48 localhost carbon-c-relay: 7f551d250000-7f551d351000 r-xp 00000000 fd:00 33990809                   /usr/lib64/libm-2.17.so
Jun 14 10:19:48 localhost carbon-c-relay: 7f551d351000-7f551d550000 ---p 00101000 fd:00 33990809                   /usr/lib64/libm-2.17.so
Jun 14 10:19:48 localhost carbon-c-relay: 7f551d550000-7f551d551000 r--p 00100000 fd:00 33990809                   /usr/lib64/libm-2.17.so
Jun 14 10:19:48 localhost carbon-c-relay: 7f551d551000-7f551d552000 rw-p 00101000 fd:00 33990809                   /usr/lib64/libm-2.17.so
Jun 14 10:19:48 localhost carbon-c-relay: 7f551d552000-7f551d573000 r-xp 00000000 fd:00 33990794                   /usr/lib64/ld-2.17.so
Jun 14 10:19:48 localhost carbon-c-relay: 7f551d6a5000-7f551d738000 rw-p 00000000 00:00 0
Jun 14 10:19:48 localhost carbon-c-relay: 7f551d769000-7f551d76d000 rw-p 00000000 00:00 0
Jun 14 10:19:48 localhost carbon-c-relay: 7f551d770000-7f551d773000 rw-p 00000000 00:00 0
Jun 14 10:19:48 localhost carbon-c-relay: 7f551d773000-7f551d774000 r--p 00021000 fd:00 33990794                   /usr/lib64/ld-2.17.so
Jun 14 10:19:48 localhost carbon-c-relay: 7f551d774000-7f551d775000 rw-p 00022000 fd:00 33990794                   /usr/lib64/ld-2.17.so
Jun 14 10:19:48 localhost carbon-c-relay: 7f551d775000-7f551d776000 rw-p 00000000 00:00 0
Jun 14 10:19:48 localhost carbon-c-relay: 7f551d776000-7f551d78e000 r-xp 00000000 fd:00 1082213                    /usr/bin/carbon-c-relay
Jun 14 10:19:48 localhost carbon-c-relay: 7f551d98d000-7f551d98e000 r--p 00017000 fd:00 1082213                    /usr/bin/carbon-c-relay
Jun 14 10:19:48 localhost carbon-c-relay: 7f551d98e000-7f551d98f000 rw-p 00018000 fd:00 1082213                    /usr/bin/carbon-c-relay
Jun 14 10:19:48 localhost carbon-c-relay: 7f551e2ad000-7f551e2ef000 rw-p 00000000 00:00 0                          [heap]
Jun 14 10:19:48 localhost carbon-c-relay: 7ffdb74f1000-7ffdb7512000 rw-p 00000000 00:00 0                          [stack]
Jun 14 10:19:48 localhost carbon-c-relay: 7ffdb75c5000-7ffdb75c7000 r-xp 00000000 00:00 0                          [vdso]
Jun 14 10:19:48 localhost carbon-c-relay: ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Jun 14 10:19:48 localhost systemd: carbon-c-relay.service: main process exited, code=killed, status=6/ABRT
Jun 14 10:19:48 localhost systemd: Unit carbon-c-relay.service entered failed state.
Jun 14 10:19:48 localhost systemd: carbon-c-relay.service failed.

It doesn't look like this is fixed in v2.0. But plase correct me if I'm wrong.

[piotr1212]

I can't reproduce this on my system. Does this happen every time or only under specific load?

[grobian]

Can you confirm it isn't fixed in v2.0?

[Civil]

It doesn't look like this is fixed in v2.0. But plase correct me if I'm wrong.

There were a lot of fixes, also including some that can cause double free in git since 1.11 (e.x. e4b6bf6 ) so most probably your issue will go away in 2.0

[piotr1212]

Ok, I can now reproduce on 1.11 but not on 2.0. it seems specific to the used configuration and version.

I will try to push 2.0 to epel-testing repos this afternoon.

[grobian]

alternatively backporting the commit referenced by @Civil should work too

[grobian]

I should release v2.1 shortly, since it includes an "include" fix, and I want to include a fix for issue #184 there too.

[bemeyert]

Oh well, You're a bit too fast for me. Thanks a lot!!!
I will try the 2.0 release.

[grobian]

fixed in v2.0

[bemeyert]

Sorry, if I didn't get back to you. But I didn't have the time to build/install from source and 2.0 isn't yet available from EPEL. Cheers

[piotr1212]

@bemeyert you can follow the tracking bug for 2.0 in Fedora and EPEL
https://bugzilla.redhat.com/show_bug.cgi?id=1341361

[bemeyert]

@piotr1212 brilliant Thx :-D