Recognize log level in Elasticsearch JVM logs

[danielmitterdorfer]

What does this PR do?

Elasticsearch will add a log level to its JVM logs to allow users to help them detect errors / warnings more easily in elastic/elasticsearch#92382. With this commit we detect this new field if present and continue to recognize the prior log format which does not contain a log level.

Why is it important?

This change is required to keep compatibility with Elasticsearch's JVM log output.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Relates elastic/elasticsearch#92382
Closes #34054

[botelastic]

This pull request doesn't have a Team:<team> label.

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @danielmitterdorfer? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[danielmitterdorfer]

There is one thing where I'm specifically looking for guidance from reviewers. In line with elasticsearch.gc.tags, I've put the log level in elasticsearch.gc.level but I wonder whether it makes more sense to put it into log.level instead.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-01-09T06:58:53.995+0000

• Duration: 72 min 28 sec

Test stats 🧪

Test	Results
Failed	0
Passed	7203
Skipped	738
Total	7941

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[danielmitterdorfer]

x-pack/filebeat-pythonIntegTest is currently failing but none of the failing tests are related to the Elasticsearch module (the failures are in cyberarkpa and cisco).

[ruflin]

Lets make sure, any change we add here also makes it into the integrations package: https://github.com/elastic/integrations/tree/main/packages/elasticsearch/data_stream/gc

[danielmitterdorfer]

Lee has already opened an issue for that: elastic/integrations#4842. :)

[leehinman]

I think log.level is the way to go, otherwise everything looks great.

[danielmitterdorfer]

I think log.level is the way to go, otherwise everything looks great.

Great, thanks for having a look @leehinman . Is this as simple as just changing the grok pattern from

(\[%{DATA:elasticsearch.gc.level}%{SPACE}\])?

to

(\[%{DATA:log.level}%{SPACE}\])?

or am I missing something?

[danielmitterdorfer]

I've pushed the rename now in 24cf0bb. @leehinman can you please take another look?

[miltonhultgren]

Lee has already opened an issue for that: elastic/integrations#4842. :)

@danielmitterdorfer When will that work be done and by whom? My team is tagged for it but it's not planned on our part.

[danielmitterdorfer]

@danielmitterdorfer When will that work be done and by whom? My team is tagged for it but it's not planned on our part.

@miltonhultgren I would hope it's not much effort once this change is in (basically copy & paste what's done here)? I'd like to see this change being released with 8.7.0. I can attempt to do this myself but will require some guidance.

[leehinman]

LGTM

[leehinman]

@danielmitterdorfer When will that work be done and by whom? My team is tagged for it but it's not planned on our part.

@miltonhultgren I would hope it's not much effort once this change is in (basically copy & paste what's done here)? I'd like to see this change being released with 8.7.0. I can attempt to do this myself but will require some guidance.

Let me know if you want help.

[miltonhultgren]

@danielmitterdorfer @leehinman Okay, I had to confirm with our expert about what would be affected but it seems like a small change so I think we can handle that, I'm putting it into our queue!

[danielmitterdorfer]

@miltonhultgren thanks for taking care of the changes on the integrations side!

@leehinman The changes are now approved and all PR checks are green but I am not authorized to merge the PR. Could you please merge the PR on my behalf? Thank you!