[Meta] Expand support for populating event.ingested

[spong]

Within Elastic Security we've recently exposed the ability for users to specify the timestamp field used when a Detection Rule runs in effort to minimize any gaps in alerts from delayed events. The ECS field most useful here would be event.ingested (elastic/ecs#453, elastic/ecs#582), and so ensuring that this field is populated whenever possible would greatly benefit any downstream use-cases where the system must determine if an event is stale or has been delayed.

Currently it looks like only two modules are setting event.ingested (thanks @leehinman!):

x-pack/filebeat/module/microsoft/defender_atp/ingest/pipeline.yml:    field: event.ingested
x-pack/filebeat/module/gsuite/ingest/common.yml:      field: event.ingested

In discussions it looks like we can add the following to relevant pipelines:

- set:
    field: event.ingested
    value: '{{_ingest.timestamp}}'

And also update the test modules here:

    # Remove event.ingested from testing, as it will never be the same.
    if obj["event.dataset"] == "microsoft.defender_atp":
        delete_key(obj, "event.ingested")
        delete_key(obj, "@timestamp")

    if obj["event.module"] == "gsuite":
        delete_key(obj, "event.ingested")

Let's use this as a meta issue for tracking the support of event.ingested across modules -- feel free to update this description as things progress.

Currently supported:

• Filebeat
  • Microsoft Defender
  • GSuite

Yet to be supported:

• TBD

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[andrewkroh]

In my opinion an optimal solution would be to have Filebeat install an Elasticsearch pipeline that adds event.ingested and then put final_pipeline into its index template to force all events through this pipeline. This would ensure that all events, not only module events, have the event.ingested value. This would take a bit of effort and this might not work so well when module transition to integration packages.

So I think the easiest way to address this would be to update the modules in bulk and add a set processor to the beginning of every module pipeline.

[mark54g]

Would this be available to all beats, if implemented, or would it be just filebeat?