Regex to split log messages

[sidali126]

https://github.com/IBM/Drain3/blob/7e340cd1eb8e27b480b0654c77e5a44a77f1331b/drain3/drain.py#L1

Hello,

is possible to add this function ?

def generate_logformat_regex(logformat):
    """ Function to generate regular expression to split log messages
    """
    headers = []
    splitters = re.split(r'(<[^<>]+>)', logformat)
    regex = ''
    for k in range(len(splitters)):
        if k % 2 == 0:
            splitter = re.sub(' +', '\\\s+', splitters[k])
            regex += splitter
        else:
            header = splitters[k].strip('<').strip('>')
            regex += '(?P<%s>.*?)' % header
            headers.append(header)
    regex = re.compile('^' + regex + '$')
    return headers, regex

This function is intriduced here : https://github.com/logpai/logparser/blob/master/logparser/Drain/Drain.py

Here is an exemple of logformat param:
log_format = '<Month> <Day> <Time> <Host> <ProcessName>: <Content>'

Then you generate the regex:
headers, regex = generate_logformat_regex(log_format)

and here is an exemple of log line:

line = 'Apr 29 06:12:56 node-elasticsearch-1 kernel: nvme nvme0: Abort status: 0x0'

Based on this regex it will be easy to get any part of the message :

match = regex.search(line)
line_content = match.group('Content')

Kind regards,
Ali

[davidohana]

Hi, its recommended that Drain3 (and also Drain afaik) is ingested with only the unstructured part of the log message ("content"). So this addition seems a bit out of context. Also it may be not robust enough to handle some more complex log formats.
Did you see the extract_parameters() function? It may suit your needs.. It also has an internal regex generation function.

[sidali126]

Hi,
thanks for your answer.
I'll try extract_parameters() and I'll keep you updated.

Kind regards,
Ali