[SPARK-49584][BUILD] Upgrade log4j2 to 2.24.0

[LuciferYang]

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

[LuciferYang]

Let's test first to confirm whether the previous deadlock issue in 2.23.1 has been resolved in this version.

[LuciferYang]

[info] - setting to MDC gets logged *** FAILED *** (13 milliseconds)
[info]   scala.Predef.augmentString(mdcPattern).r.findFirstIn(logOutputWithMDCSet).isDefined was false (StructuredLoggingSuite.scala:473)
[info]   org.scalatest.exceptions.TestFailedException:
[info]   at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
[info]   at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
[info]   at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
[info]   at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
[info]   at org.apache.spark.util.StructuredLoggingSuite.$anonfun$new$8(StructuredLoggingSuite.scala:473)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
[info]   at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
[info]   at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
[info]   at org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1564)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
[info]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
[info]   at org.scalatest.funsuite.AnyFunSuite.runTest(AnyFunSuite.scala:1564)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
[info]   at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
[info]   at scala.collection.immutable.List.foreach(List.scala:334)
[info]   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
[info]   at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
[info]   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
[info]   at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
[info]   at org.scalatest.Suite.run(Suite.scala:1114)
[info]   at org.scalatest.Suite.run$(Suite.scala:1096)
[info]   at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
[info]   at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
[info]   at org.scalatest.funsuite.AnyFunSuite.run(AnyFunSuite.scala:1564)
[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info]   at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info]   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info]   at java.base/java.lang.Thread.run(Thread.java:840)

A test has failed and needs to be investigated.

[LuciferYang]

https://github.com/apache/spark/blob/7355864e2cf8329188e69365415ab0bd718d0c1a/common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala#L468C3-L478

  test("setting to MDC gets logged") {
    val mdcPattern = s""""${LogKeys.DATA.name}":"some-data""""

    org.slf4j.MDC.put(LogKeys.DATA.name, "some-data")
    val logOutputWithMDCSet = captureLogOutput(() => logInfo(msgWithMDC))
    assert(mdcPattern.r.findFirstIn(logOutputWithMDCSet).isDefined)

    org.slf4j.MDC.remove(LogKeys.DATA.name)
    val logOutputWithoutMDCSet = captureLogOutput(() => logInfo(msgWithMDC))
    assert(mdcPattern.r.findFirstIn(logOutputWithoutMDCSet).isEmpty)
  }

There is a significant refactoring in the ThreadContextMap component(apache/logging-log4j2#2330), resulting in the loss of key-value pairs injected via org.slf4j.MDC during the generation of LogEvent objects when using DefaultThreadContextMap. As a workaround, it is necessary to configure and employ GarbageFreeSortedArrayThreadContextMap. I am puzzled by this situation. Since org.slf4j.MDC is a behavior specific to SLF4J, and this does not appear to be an incompatible change for Log4j, let's currently halt this upgrade and wait to see if the new version of SLF4J can address this issue.

[LuciferYang]

cc @dongjoon-hyun and @panbingkun for awareness

[dongjoon-hyun]

cc @gengliangwang , too

[dongjoon-hyun]

BTW, @LuciferYang , there is a PR already from @panbingkun .

• [SPARK-49541][BUILD] Upgrade log4j2 to 2.24.1 #48029

[LuciferYang]

ok, let me close this one :)