HIVE-28417: Bump Log4j2 to 2.24.1 to facilitate compilation of GraalVM Native Image

[linghengqian]

What changes were proposed in this pull request?

• Bump Log4j2 to 2.24.1 to facilitate compilation of GraalVM Native Image.

Why are the changes needed?

• This PR is aimed at ensuring that HiveServer2 JDBC Driver is compatible with future versions of LOG4J as much as possible. I created a unit test at https://github.com/linghengqian/log4j2-v2231-graalvm-native-image-test to prove that the HiveServer2 JDBC Driver using apache/logging-log4j2:2.23.1 can be used under the GraalVM Native Image compiled by GraalVM CE For JDK 22.0.2.

sdk install java 8.0.422-tem
sdk install java 22.0.2-graalce
sdk use java 8.0.422-tem
sdk install maven
git clone [email protected]:linghengqian/hive.git -b log4j-bump
cd ./hive/
mvn clean install -DskipTests
mvn clean package -pl packaging -DskipTests -Pdocker
cd ../

git clone [email protected]:linghengqian/log4j2-v2231-graalvm-native-image-test.git
cd ./log4j2-v2231-graalvm-native-image-test/
sdk use java 22.0.2-graalce
./mvnw -PgenerateMetadata -DskipNativeTests -e -T1C clean test native:metadata-copy
./mvnw -PnativeTestInHive -T1C -e clean test

• See also https://github.com/apache/logging-log4j2/blob/rel/2.23.1/src/changelog/2.20.0/LOG4J2-3650_Move_tests.xml .

Moved `log4j-api` and `log4j-core` artifacts with classifier `tests` to `log4j-api-test` and `log4j-core-test` respectively. 

• See also log4j-core test-jar not published to Maven Central for 2.20.0 release logging-log4j2#1291 (comment) .

I took care of the [release notes](https://logging.apache.org/log4j/2.x/release-notes/2.20.0.html).

The difference in dependencies is caused by Maven:

for the log4j-core artifact with tests classifier Maven pulled the wrong dependencies (compile scope instead of test scope),
for the log4j-core-test artifact the real dependencies are no longer hidden: the artifact mostly contains JUnit 4 and JUnit 5 extensions, so it depends on all those libraries.

• See also Sync SLF4J 1.x and 2.x bindings logging-log4j2#1032 . The constructor of org.apache.logging.slf4j.Log4jMarker has changed.
• See also Fix handling of onMatch and onMismatch attributes in the properties configuration format logging-log4j2#2791 and When bump from 2.23.0 to 2.23.1, the status of the .properties configuration file no longer works logging-log4j2#2794 . The Log4j configuration file used by Hive contains deprecated properties .

Does this PR introduce any user-facing change?

No.

Is the change a dependency upgrade?

• Below is the output of mvn dependency:tree > ./dependency-tree.txt.

• dependency-tree.txt

How was this patch tested?

sdk install java 8.0.422-tem
sdk use java 8.0.422-tem
sdk install maven
mvn clean install -DskipTests
mvn test -Dtest=TestHplsqlLocal -pl hplsql
mvn test -Dtest=TestHplsqlOffline -pl hplsql

[linghengqian]

• @pgaref I don't understand why the bot didn't update the github tag after I put the PR back for review. Do you know how to make the bot update the github tag again?
• update 1st: To be honest, I can't understand the error log of https://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-5375/5/tests .

Unexpected exception java.lang.RuntimeException: Process docker failed to run in 30 seconds
 at org.apache.hadoop.hive.ql.externalDB.AbstractExternalDB.runCmd(AbstractExternalDB.java:91)
 at org.apache.hadoop.hive.ql.externalDB.AbstractExternalDB.runCmdAndPrintStreams(AbstractExternalDB.java:106)

• update 2nd: I took a moment to get to know the Jenkins UI... Looks like it should work, right?
• update 3rd: Wait Fix handling of onMatch and onMismatch attributes in the properties configuration format logging-log4j2#2791 .
• update 4th: All done.

[Aggarwal-Raghav]

Had 1 question, added it as comment. Otherwise java and pom changes LGTM (+1 non-binding).

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the [email protected] list if the patch is in need of reviews.

[linghengqian]

Due to the bugs https://issues.apache.org/jira/browse/HIVE-28559 and https://issues.apache.org/jira/browse/HIVE-28552, I can't actually merge the master branch directly.

[zhangbutao]

Why this changed?

[linghengqian]

• According to Fix handling of onMatch and onMismatch attributes in the properties configuration format logging-log4j2#2791, we no longer need such list definition, log4j2 will scan it automatically. Not surprisingly, retaining this attribute causes the warning.

2024-08-02T10:55:16.634Z main WARN The use of package scanning to locate plugins is deprecated and will be removed in a future release
2024-08-02T10:55:16.650Z main WARN The use of package scanning to locate plugins is deprecated and will be removed in a future release
2024-08-02T10:55:16.670Z main WARN The use of package scanning to locate plugins is deprecated and will be removed in a future release
2024-08-02T10:55:16.686Z main WARN The use of package scanning to locate plugins is deprecated and will be removed in a future release

[zhangbutao]

https://web.archive.org/web/20240823014436/https://logging.apache.org/log4j/2.x/manual/configuration.html#configuration-syntax

Use of the packages attribute is deprecated and will be removed in Log4j 3.0. Plugins should be processed with the Log4j annotation processor. A comma separated list of package names to search for plugins. Plugins are only loaded once per classloader so changing this value may not have any effect upon reconfiguration.

How we replace packages with the new way? I think removing package will cause some unkown issue.

[linghengqian]

How we replace packages with the new way? I think removing package will cause some unkown issue.

• I would say that even without merging this PR, packages never worked in Hive. Unless Hive has always used log4j1 and never used log4j2. Because the class org.apache.logging.log4j.core.config.plugins.processor.PluginProcessor exists in the JAR of the final binary of Hive, org.apache.logging.log4j.core.config.plugins.processor.PluginProcessor generates Log4j2Plugins.dat, which directly covers the operation of packages. It is located at /hive/ql/target/hive-exec-4.1.0-SNAPSHOT.jar!/META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat. packages is actually a log4j1 thing.
• 
• You can determine from a comment by a log4j2 committer that this property does not actually take effect. See Fix handling of onMatch and onMismatch attributes in the properties configuration format logging-log4j2#2791 (comment) .

There is no problem in Apache Hive: I checked that the appropriate annotation processor is configured in the hive-exec project and there is a Log4j2Plugins.dat file in the JAR. You can safely remove the packages configuration attribute.

• org.apache.logging.log4j.core.config.plugins.processor.PluginProcessor is available since log4j2 2.0. The processor will generate the necessary META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat descriptor that replaces classpath scanning. See After modularization of Log4j2, loading the plugin (PluginManager. addPackage) fails logging-log4j2#2434 (comment) . There is also an intuitive PR at Removes the deprecated packages Log4j configuration attribute jmeter#6326 .

• Since Hive has neither CI for JDK23 and JDK24 nor CI for GraalVM Native Image, we actually don't need to register a org.apache.logging.log4j.core.config.plugins.processor.PluginProcessor in pom.xml as https://logging.apache.org/log4j/2.x/manual/plugins.html#plugin-registry .

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-compiler-plugin</artifactId>
  <version>${maven-compiler-plugin.version}</version>
  <executions>
    <execution>
      <id>generate-log4j-plugin-descriptor</id>
      <goals>
        <goal>compile</goal>
      </goals>
      <phase>process-classes</phase>
      <configuration>
        <proc>only</proc>
        <annotationProcessorPaths>
          <path>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>2.24.1</version>
          </path>
        </annotationProcessorPaths>
        <annotationProcessors>
          <processor>org.apache.logging.log4j.core.config.plugins.processor.PluginProcessor</processor>
        </annotationProcessors>
      </configuration>
    </execution>
  </executions>
</plugin>

• You may ask why. The reason is simple, because for JDK8-JDK22, the default value of the JVM parameter -proc is full. The JVM parameter -proc did not become only until JDK23. -proc of full means that all processors are included by default. See Support compiling and using ShardingSphere under OpenJDK 23 shardingsphere#33025 and OpenJDK Committer’s Blog https://inside.java/2024/06/18/quality-heads-up/ . I also suspect that the ASF Jenkins node has an outdated JDK8 version and does not actually recognize the -proc JVM parameter, just like the Temurin JDK 8 on the Github Actions Runner also does not recognize the -proc JVM parameter.
• Are there unit tests for Hive Exec under GraalVM Native Image compiled by GraalVM CE For JDK22 and GraalVM CE For JDK23? Yes, they are at https://github.com/linghengqian/hive-server2-jdbc-driver and https://github.com/apache/shardingsphere . Due to the ongoing dispute of https://issues.apache.org/jira/browse/HIVE-28445, I have to maintain Hive's Thin JAR outside of Hive to provide it for unit testing of ShardingSphere under GraalVM Native Image.
• Does packages = org.apache.hadoop.hive.ql.log affect things like logger.Operator.name = org.apache.hadoop.hive.ql.exec.Operator? Apparently not. Most people use packages because they are actually using log4j1.

[zhangbutao]

Good explanation!

Since Hive has neither CI for JDK23 and JDK24 nor CI for GraalVM Native Image, we actually don't need to register a org.apache.logging.log4j.core.config.plugins.processor.PluginProcessor in pom.xml as https://logging.apache.org/log4j/2.x/manual/plugins.html#plugin-registry .

BTW, if Hive wants to use JDK23 or JDK24, should we need to register the PluginProcessor ?

[linghengqian]

BTW, if Hive wants to use JDK23 or JDK24, should we need to register the PluginProcessor ?

• The answer is yes. But I think most people will actually choose to set -proc:full for JDK 23 to make JDK 23 behave the same as JDK8. -proc:full is only available on 11.0.23, 17.0.11, JDK 21 - 24. -proc:full is not available on any OpenJDK 8u, -proc:full is only available on Oracle's closed source JDK 8u411.
• For JDK23, we can set <maven.compiler.proc>full</maven.compiler.proc> in a separate Maven Profile to make JDK23 behave the same as JDK 8. Refer to Support building Example module with OpenJDK 23 shardingsphere#33224 for the solution.
• I would say setting <maven.compiler.proc>full</maven.compiler.proc> is not useful for Hive, as I found that the master branch of Hive cannot be compiled with JDK22. This means that none of the unit tests will be affected.

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[zhangbutao]

+1 LGTM

Thanks for the patch!