[Filebeat] [Processors] Add Append Processor

[mr1716]

This adds an append processor to be able to append data to ECS values.

The use case for this specific enhancement is that the majority of filebeat modules already contain the append processor at least 1 time in the ECS Modules, and this would allow beat-side processing to occur rather than on the Elastic side.
Team: Filebeat

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[mr1716]

@mtojek Thanks for adding a label. Would it also be worth adding the enhancement and Filebeat labels in addition to the existing label?

[mr1716]

Just for additional context, this is one of the more popular elastic processors seen in Filebeat Pipelines, so it will be great to get this ported into Filebeat. It is seen in 52 modules, out of 70 modules in Filebeat, which puts it at 74.285% of the modules. This will make the append processor the 4th most used elastic processor for Filebeat modules. There are already applicable native Filebeat processors for #s 1-3, which are set, rename, and remove. @jlind23

[mr1716]

Hey, I wanted to ping on this to see if there could be some traction on this going forward

[nimarezainia]

@mr1716 are you experiencing performance issues and want to move this processing out to the beat? Just trying to understand the use case and what you are trying to achieve. thanks.

[mr1716]

@nimarezainia I would like to move this to the beat to make sure that it can handle the data properly and provide parity for my use case.

[nimarezainia]

@mr1716 could you explain the use case in a bit more detail? I'm curious why doing this at the Elastic side is not handling the data properly?

[mr1716]

@nimarezainia right now, if I wanted to append data to a field in filebeat, I would have to do this via script. I am trying to reduce the work that the backend does. Therefore, I would like to have more parity and help spread out the workload.

[BenB196]

I've been following this request and I think a good use case for append to is the setting of the *.address at the beats level. Today, unless I'm mistaken you can't easily add both the *.domain and *.ip values to the *.address field (I think it might be possible with a script processor, but that seems very clunky (and isn't officially documented)).

A good reason for why I'd want to do this type of thing at the beat level, rather than Logstash or Elasticsearch Ingest level, is a lot of times it isn't really needed, and all using those methods does is add additional complexity. If I can do 95% of what I need to do at the beats level, why add an additional "thing" to the process to do the last 5%, when it would be better to keep it at the beats level.

A point worth noting, is that the concept of append isn't really new to beats, the DNS process already supports this under its action as the default value.

Another note, is that currently this is only being looked at if you're outputting to Elasticsearch or Logstash, but Beats have a variety of support outputs, so being able to modify the data as needed to these other outputs would also provide value.

[UcanInfosec]

@nimarezainia ive also been watching this and would love this to get added. Any thoughts on when this could be added? Adding this would be extremely beneficial to our grater adoption of the tool

[mr1716]

Hey, I dont see the documentation for that here: https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html