Filters/macros/function for Vector's template syntax

[binarylogic]

This has been discussed in a variety of issues (...), but we don't have an actual issue open for this work.

Motivation

Our template syntax is very limited in its current form and we've had a number of customers request ways to extend this (#1846, #1692, #1440, #1926 (comment))

Prior Art

1. Liquid filters - Rust implementation
2. Jinja 2 filters

Proposal

We should support "modifiers" or "filters" within our template syntax. For example:

[sinks.archive]
  type = "aws_s3"
  key_prefix = "{{ timestamp | date: "%Y-%m-%d" }}"

Filters

We can use the prior art as a basis, but at the very least we should have:

• default
• strptime
• capitalize
• downcase
• truncate
• upcase

These are most useful for dynamic partitioning.

[Hoverbear]

We could use tera for this! https://tera.netlify.com/ It's basically jninja or the Django language.

[lukesteensen]

The projects I'd flagged as worth investigating for this were, in no particular order:

• https://github.com/sunng87/handlebars-rust
• https://github.com/cobalt-org/liquid-rust
• https://github.com/Keats/tera
• https://github.com/proctorlabs/templar

[Hoverbear]

I've been using Tera through Zola for my website and it feels very decent. Vector (given our erb templates for our website) might prefer liquid though!

[modularinput]

I would love to see Vector offer something that was more Linux Shell meets SQL (see below). If this capability were available it could unify template and transform syntax.

Example

[transforms.example]
type = "Universal Processing Language"
command = [
from source_X OR tranform_Y
| evaluate key = len(key) 
| evaluate key = if(isnull(key), "true", "false") 
| evaluate key = if(cidrmatch(key, "192.168.0.0/24"), "internal", "external")
| fields - timestamp, message, source 
| evaluate key = lower(key)
| evaluate key =strftime(key, "%F %T")
| regex source_key = "^.*?(?<new_key>capture_this)" ]

[binarylogic]

Hi @modularinput, have you seen the Vector Remap Language that we're planning to release in 0.12? That should fill the void here.

[modularinput]

Hello @binarylogic. Thanks for pointing this out.

Here are my first impressions after reviewing the docs.

1. I really like the direction you are taking this.
2. I hope there is a possibility to drop the . syntax.
3. I do not find the first line see quoted code to be intuitive at all, but this is probably just me.

It is clear I need to read more and try it out first. Just wanted to give you my initial thoughts.

Vector Remap Language

. |= parse_common_log!(.log)
.total_bytes = .bytes_in + .bytes_out
del(.log)
del(.time)

Reference 2

I think the following would be easier to read and write.

total_bytes = bytes_in + bytes_out
del(log)
del(time)

Reference 3

I don't have quality feedback for this yet, but hope to after giving it a try.

This is what I think I understand from the first read over:

• parse_common_log is a prepackaged parser for a certain type of structured / formatted data.
• log is the key we are working with.
• ! is some sort of error handling.
• . |= I have no idea.