Filebeat Log Collection Halting : ERR Fail to convert the event to JSON: reflect: call of reflect.Value.IsNil on zero Value

[david-rooke]

I am seeing that log collection is halting periodically when sending Datastax Enterprise Cassandra logs to logstash. Issue was originally seen in version 1.1.0 and 1.2.1 with this error:

ERR Fail to convert the event to JSON: reflect: call of reflect.Value.IsNil on zero Value

Installing a 5.0.0 nightly build didn't resolve the problem or provide any additional output. Indeed, the service tends to die silently...

Additional information and background can be found here (https://discuss.elastic.co/t/filebeat-1-1-0-log-collection-halting/47130/15)

• Version: 1.1.0, 1.2.1, 5.0.0 (currently using 5.0.0 nightly)
• Operating System: Ubuntu 14.04
• Steps to Reproduce: I have attached a sample sanitised log from the server and filebeat shipper.

sample_logs.tar.gz

[tsg]

@david-rooke can you also provide your filebeat.yml? I know you posted it in discuss, but I want to make sure we have your up-to-date version before we try to reproduce this. Thanks!

[david-rooke]

@tsg config attached...
filebeat_conf.tar.gz

[tsg]

@david-rooke we have a suspicion that the multiline implementation might be causing this, but in the sample logs you provided there doesn't seem to be any multiline events (lines starting with a white space).

1. If you can reproduce this relatively easily, can you try removing the multiline configuration to confirm this theory.
2. Can you provide a sample log that contains such a multiline event?

Thanks!

[tsg]

Ah, nevermind point 2, I see now that it contains an exception, it was just my regexp being broken. Point 1 could still be interesting.

[david-rooke]

Well, I've seen this behaviour whether multiline has been enabled or not.

[tsg]

Thanks, that's useful information.

[tsg]

I managed to reproduce the issue. It's related to the fact that there are two outputs enabled (console and logstash). It can happen that they try to modify the same event concurrently which results in a panic. Stack trace here: https://gist.github.com/tsg/fbfdac8ea8a0b1f74f17e4a7c4715dee

We probably need to do deep copying of the events before passing them to multiple outputs.

@david-rooke As a workaround, I'd recommend disabling the console output for the moment.

[urso]

True, the logstash output adds meta-data to the event without copying it. potential race on map.