logparser does not appear to parse BASE16NUM

[thermionic]

Bug report

Using BASE16NUM in a logparser pattern appears to result in logparser not parsing the log line

Relevant telegraf.conf:

[[inputs.logparser]]
files = ["/home/username/ptpd2.stats"]
from_beginning = true
[inputs.logparser.grok]
patterns = ["%{PTPD2}"]
measurement = "ptpd2"
custom_patterns = '''
PTPD2 %{TIMESTAMP_ISO8601:utc:ts-"2006-01-02 15:04:05.000000"},%{SPACE}%{WORD:state:tag},%{SPACE}%{BASE16NUM:grandmasterMAC:tag}

[[outputs.file]]
files = ["stdout"]

System info:

Ubuntu 16.04
Telegraf 1.12

Steps to reproduce:

1. telegraf --config telegraf.conf

Expected behavior:

I! Output [file] buffer fullness: 3 / 10000 metrics. Total gathered metrics: 3. Total dropped metrics: 0.

Actual behavior:

I! Output [file] buffer fullness: 0 / 10000 metrics. Total gathered metrics: 0. Total dropped metrics: 0.

Additional info:

As for this application the string (actually a MAC address) is only being used as a tag, the workaround was to use USERNAME instead of BASE16NUM, but this bug report is filed to make aware of teh BASE16NUM issue

With the modifier (":tag" etc) removed, the pattern tests without issue on http://grokconstructor.appspot.com/do/match

fragment of log file
2017-01-21 01:17:47.761598, slv, b827ebfffe6caf1c(unknown)/1, 0.000145828, 0.000917194, -0.000731000, 0.000961000, -99635.728000000, D, 01435, 0.000145570, 841, 0.001342567, 120658, -139135, 11777, 0.000961000, -0.000731000
2017-01-21 01:17:57.429593, slv, b827ebfffe6caf1c(unknown)/1, 0.000143496, 0.000799778, -0.000647000, 0.001013000, -92054.901000000, D, 01443, 0.000146656, 1516, 0.000951777, 97928, -105084, 8306, 0.001013000, -0.000647000
2017-01-21 01:17:57.541595, slv, b827ebfffe6caf1c(unknown)/1, 0.000143496, 0.000790010, -0.000647000, 0.000854000, -92054.901000000, S, 61929, 0.000146656, 1516, 0.000951777, 97928, -105084, 8306, 0.000854000, -0.000647000

[thermionic]

I think I have found the cause.

https://github.com/influxdata/telegraf/blob/master/plugins/inputs/logparser/grok/grok.go
appears to pull
https://github.com/vjeantet/grok/blob/master/patterns/grok-patterns
which contains
BASE16NUM (0[xX]?[0-9a-fA-F]+)
which does not match
BASE16NUM (?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+))
from the source
https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns

[sparrc]

thanks for looking into that, do you mind submitting a PR to fix it?

[thermionic]

I had a go fixing it, but possibly because of an artefact of the build system, I wasn't able to get it to pass my basic tests...

[sparrc]

ah, that may be because Go doesn't support regex lookaheads or lookbehinds, such as ?<

[sparrc]

dupe of #2178