Extend the length of OpenSearchJsonLayout logger from 10k to 100k

[djboris9]

Description

Especially audit logs from OpenSearch can be longer than 10k characters. If Log4J is configured with the OPENSEARCHJsonLayout pattern type, the inner message (which is a json) tends to get truncated and therefore the deserialization of the inner message fails and leads to missing information. Depending on the logs, this is not what a customer would expect to happen.

Therefore I propose to extend the truncation from 10k characters to at least 100k characters but I'm also happy if it's extended further. This allows for longer log messages and therefore should serialize much more logs correctly but potentially increases the log file growth in cases, where the messages were regularely truncated.

Example log4j2.properties:

appender.audit_rolling.type = RollingFile
appender.audit_rolling.name = audit_rolling
appender.audit_rolling.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}_audit.json
appender.audit_rolling.layout.type = OPENSEARCHJsonLayout
appender.audit_rolling.layout.type_name = audit
appender.audit_rolling.layout.opensearchmessagefields=x-opaque-id

Example of truncated logs, anonymized and shortened:

{
  "type": "audit",
  "timestamp": "2023-03-07T01:01:01,000+00:00",
  "level": "INFO",
  "component": "audit",
  "cluster.name": "MASKED",
  "node.name": "MASKED_DATA",
  "message": "{\"audit_trace_task_parent_id\":\"MASKED\",\"audit_cluster_name\":\"MASKED\",....\".kibana_20353",  <-- This end is truncated
  "cluster.uuid": "MASKED",
  "node.id": "MASKED"
}

Issues Resolved

No issue was created

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/12083/
• CommitID: f7a177b
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/12082/
• CommitID: 5857f67
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/12104/
• CommitID: 1c46022

[codecov-commenter]

Codecov Report

Merging #6569 (1c46022) into main (8e0edee) will increase coverage by 0.14%.
The diff coverage is 0.00%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main    #6569      +/-   ##
============================================
+ Coverage     70.65%   70.80%   +0.14%     
- Complexity    59075    59204     +129     
============================================
  Files          4804     4804              
  Lines        283075   283075              
  Branches      40806    40806              
============================================
+ Hits         200013   200436     +423     
+ Misses        66683    66160     -523     
- Partials      16379    16479     +100     

Impacted Files	Coverage Δ
...pensearch/common/logging/OpenSearchJsonLayout.java	0.00% <0.00%> (ø)
...main/java/org/opensearch/node/NodeMocksPlugin.java	0.00% <0.00%> (-100.00%)	⬇️
.../index/shard/IndexShardNotRecoveringException.java	0.00% <0.00%> (-50.00%)	⬇️
.../org/opensearch/client/indices/AnalyzeRequest.java	31.00% <0.00%> (-45.00%)	⬇️
...opensearch/persistent/PersistentTasksExecutor.java	22.22% <0.00%> (-44.45%)	⬇️
...ava/org/opensearch/threadpool/ThreadPoolStats.java	52.00% <0.00%> (-29.34%)	⬇️
...a/org/opensearch/index/mapper/MapperException.java	75.00% <0.00%> (-25.00%)	⬇️
...va/org/opensearch/monitor/process/ProcessInfo.java	68.00% <0.00%> (-24.00%)	⬇️
...nsearch/index/shard/IndexingOperationListener.java	63.93% <0.00%> (-22.96%)	⬇️
...pensearch/index/mapper/MapperParsingException.java	77.77% <0.00%> (-22.23%)	⬇️
... and 500 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[reta]

👍 , do you think the upper limit would also make sense to let say 500k? (going with 2g messages is too much I think)

[djboris9]

I was also thinking about to enforce a sane upper limit. But in the end the user still has the ability to set the maxmessagelength = 500000. On the other side, I can imagine that the integrity/completeness of audit logs are for many customers important and therefore some hard upper limit isn't desirable. This way I went for a explicit unrestricted limit you can chose with maxmessagelength = 0 and keeping the default at 10'000 chars

[reta]

Could you please move the changelog entry to ## [Unreleased 2.x], we'll backport it to 2.x branch as well, thank you

[djboris9]

Done with 227fdfc

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationDuringQueryPhase

• URL: https://build.ci.opensearch.org/job/gradle-check/12169/
• CommitID: 1de9ee6
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/12170/
• CommitID: 2e71a6e
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[djboris9]

@reta Should I squash my commits before it hopefully gets merged?

[reta]

@reta Should I squash my commits before it hopefully gets merged?

Up to you, the change is small enough (and it will be squashed on merge)

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/12171/
• CommitID: 227fdfc

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testReadNonexistentBlobThrowsNoSuchFileException

• URL: https://build.ci.opensearch.org/job/gradle-check/12172/
• CommitID: f9668cd
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[reta]

@andrross any concerns regarding this change? thank you