Skip to content
This repository has been archived by the owner on Sep 21, 2023. It is now read-only.

Migrate the default global Beat processors to the shipper #292

Open
Tracked by #16
cmacknz opened this issue Apr 4, 2023 · 1 comment
Open
Tracked by #16

Migrate the default global Beat processors to the shipper #292

cmacknz opened this issue Apr 4, 2023 · 1 comment
Labels
Team:Elastic-Agent Label for the Agent team

Comments

@cmacknz
Copy link
Member

cmacknz commented Apr 4, 2023

The agent currently configures a set of default processors to apply globally to all events collected from each Beat. See elastic/beats#34149 for the original implementation.

Several of these processors are expensive to create and will cache copies of the same metadata in each Beat they run in. See elastic/beats#35000 for an example of a performance issue introduced by creating too many instances of these processors.

We should migrate these processors into the shipper to minimize the number of instances of them that need to be run, and allow them to apply to all data collected by agent.

One challenge will be that the default processor configuration for each beat is almost but not quite the same. We will need to evaluate whether these differences are significant enough to warrant apply the processors for all data routed through the shipper or only for data routed through the shipper from a specific Beat.

@cmacknz
Copy link
Member Author

cmacknz commented May 5, 2023

Copying my comment from https://github.com/elastic/beats/pull/35318/files#r1186277241

The complexity to this comes from:

  1. If someone disables the shipper the default processors need to go back in the inputs
  2. The default processor configurations vary somewhat by Beat
  3. The shipper's default processors are true global processors and will apply to events the did not before, like endpoint security.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Team:Elastic-Agent Label for the Agent team
Projects
None yet
Development

No branches or pull requests

1 participant