Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[filebeat][elasticsearch] deprecation, server and slowlog filesets use the wrong subpipeline version #39802

Closed
miltonhultgren opened this issue Jun 4, 2024 · 1 comment
Labels
Filebeat Filebeat Module:elasticsearch Elasticsearch Beats modules Team:Monitoring Stack Monitoring team

Comments

@miltonhultgren
Copy link
Contributor

Since stack version 8:
deprecation, server and slowlog have supported two log formats.
For version 7 of the stack we used a non-ECS JSON format and for version 8 it's a ECS JSON format.

If you look at the 3 pipelines for how they branch into the version 7 or version 8 sub-pipelines, it's all based on if the ecs.version is part of the incoming message.

Current pipeline code:

ctx.message.contains("ecs.version")'

But this doesn't work because ecs.version isn't part of the ECS formatted message that Elasticsearch is outputting.

Compare this to what does work:

ctx.containsKey("ecs.version")'

(since ecs.version is written to the root of the ingested document by the add_fields processor here).

We should update these 3 pipelines to make sure that version 7 goes to the version 7 pipeline and version 8 goes to the version 8 pipeline correctly.

@miltonhultgren miltonhultgren added Filebeat Filebeat Module:elasticsearch Elasticsearch Beats modules Team:Monitoring Stack Monitoring team labels Jun 4, 2024
@miltonhultgren
Copy link
Contributor Author

The pipelines are fine, it seems to be an issue in ESS that the log format is wrong.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Filebeat Filebeat Module:elasticsearch Elasticsearch Beats modules Team:Monitoring Stack Monitoring team
Projects
None yet
Development

No branches or pull requests

1 participant