GH-41307: [Java] Use org.apache:apache parent pom version 31 #41772

laurentgo · 2024-05-22T05:14:02Z

Use/update Maven modules to org.apache:apache:31 and clean up Maven modules to remove unnecessary configuration or outdated workarounds

Add org.apache:apache:31 to org.apache.arrow:arrow-bom and org.apache.arrow.maven.plugins:arrow-maven-plugins to make them conformant with ASF standards
Update org.apache.arrow:arrow-java-root parent to org.apache:parent:31
Use version.* and other properties to override plugin versions defined by org.apache:parent
Move standalone plugin versions under pluginManagement at the top level
Cleanup redundant plugin version or configuration declaration
Update maven-dependency-plugin to 3.6.1 and add the required overrides when necessary
Update maven-shade-plugin to 3.5.1 (via org.apache:parent)
- disable reduced dependency pom creation for non-terminal modules
Remove enforcer check for java and maven version (handled by org.apache:parent)
Remove unnecessary mvnrepository link comments
Remove m2e.version property check in profiles (only needed for errorprone plugin configuration which is incompatible with M2E)
Cleanup argLine overrides for surefire/failsafe plugins
Remove unnecessary ../pom.xml <relativePath> directives
Remove source/target/encoding configuration properties for maven-compiler-plugin, maven-javadoc-plugin and maven-resources-plugin as it is handled by org.apache:parent and plugin themselves
Remove unnecessary copy of codegen templates in arrow-vector module
Remove unnecessary junit jupiter engine dependencies for surefire/failsafe plugins.
GitHub Issue: [Java] Update Maven modules to use latest Apache Parent (31) and plugins #41307

laurentgo · 2024-05-22T05:24:14Z

@lidavidm re-uploading the pull request with the fix for maven-shade-plugin behavior. Are you aware of any other issue caused by this change reported since the migration? (I checked the current issues but haven't seen anything)

laurentgo · 2024-05-22T05:27:41Z

@github-actions crossbow submit java test-conda-python-3.10-spark-v3.5.0

github-actions · 2024-05-22T05:28:19Z

Only contributors can submit requests to this bot. Please ask someone from the community for help with getting the first commit in.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/9185840180

laurentgo · 2024-05-22T05:40:45Z

The Java JNI / AMD64 Conda Java C Data Interface Integration (pull_request) issue seems to be related to #41725

vibhatha · 2024-05-22T10:20:06Z

@github-actions crossbow submit -g java

github-actions · 2024-05-22T10:22:27Z

Revision: a5c8d7d

Submitted crossbow builds: ursacomputing/crossbow @ actions-b536a3215e

Task	Status
java-jars
test-conda-python-3.10-spark-v3.5.0
test-conda-python-3.11-spark-master
test-conda-python-3.8-spark-v3.5.0
verify-rc-source-java-linux-almalinux-8-amd64
verify-rc-source-java-linux-conda-latest-amd64
verify-rc-source-java-linux-ubuntu-20.04-amd64
verify-rc-source-java-linux-ubuntu-22.04-amd64
verify-rc-source-java-macos-amd64

vibhatha · 2024-05-22T12:38:09Z

@github-actions crossbow submit java test-conda-python-3.10-spark-v3.5.0

These seems to be passing, but there are some other CIs failing.

laurentgo · 2024-05-22T12:48:31Z

Yes, noticed that some flight tests seem flaky? I also found occurrence of this issue in other pull requests like this job for example: https://github.com/apache/arrow/actions/runs/9185953424/job/25260768750

vibhatha · 2024-05-22T12:50:49Z

Yes, noticed that some flight tests seem flaky? I also found occurrence of this issue in other pull requests like this job for example: https://github.com/apache/arrow/actions/runs/9185953424/job/25260768750

Yes indeed. Would you mind filing an issue for that?

laurentgo · 2024-05-23T16:16:28Z

@vibhatha it looks like I may need to incorporate some of the work you did for #41586 ?

laurentgo · 2024-05-23T17:15:28Z

@vibhatha would appreciate if you could trigger another crossbow run to validate latest changes

jbonofre · 2024-05-23T19:29:28Z

It should fix the problem I mentioned in another issue. Thanks !

vibhatha · 2024-05-23T23:37:01Z

@github-actions crossbow submit -g java

github-actions · 2024-05-23T23:39:28Z

Revision: 37f0b16

Submitted crossbow builds: ursacomputing/crossbow @ actions-a6201b76d3

Task	Status
java-jars
test-conda-python-3.10-spark-v3.5.0
test-conda-python-3.11-spark-master
test-conda-python-3.8-spark-v3.5.0
verify-rc-source-java-linux-almalinux-8-amd64
verify-rc-source-java-linux-conda-latest-amd64
verify-rc-source-java-linux-ubuntu-20.04-amd64
verify-rc-source-java-linux-ubuntu-22.04-amd64
verify-rc-source-java-macos-amd64

vibhatha · 2024-05-23T23:39:41Z

@vibhatha would appreciate if you could trigger another crossbow run to validate latest changes

I didn't complete it, just tried a few ideas.

vibhatha · 2024-05-28T14:48:37Z

@laurentgo we have an issue in the dataset module: https://github.com/ursacomputing/crossbow/actions/runs/9216288007/job/25356948996#step:6:55162

laurentgo · 2024-05-28T14:57:04Z

I noticed but I'm not able to reproduce at this time. Looks like there's been changes to java/pom.xml causing a conflict, so will rebase and force push. Would appreciate if someone could trigger a crossbow run just after

vibhatha · 2024-05-28T15:11:22Z

@github-actions crossbow submit -g java

vibhatha · 2024-06-05T11:33:31Z

Let's rebase and try again. Unfortunately, that boot error in tests happens occasionally and we don't have a good way of debugging it.

@vibhatha @danepitkin I think it would be a good improvement to have these jobs collect JVM dump files in the case of failure so we can try to debug this properly.

Yes, @lidavidm we should. I created an issue #41982

vibhatha · 2024-06-05T11:38:33Z

Here's the output of the jvm dumpstream file after 15 attempts:

# Created at 2024-06-05T08:41:26.092
Corrupted channel by directly writing to native stream in forked JVM 2. Stream '#'.

# Created at 2024-06-05T08:41:26.092
Corrupted channel by directly writing to native stream in forked JVM 2. Stream '# A fatal error has been detected by the Java Runtime Environment:'.

# Created at 2024-06-05T08:41:26.092
Corrupted channel by directly writing to native stream in forked JVM 2. Stream '#'.

# Created at 2024-06-05T08:41:26.094
Corrupted channel by directly writing to native stream in forked JVM 2. Stream '#  SIGSEGV (0xb) at pc=0x000000012900f0b0, pid=12023, tid=25859'.

# Created at 2024-06-05T08:41:26.094
Corrupted channel by directly writing to native stream in forked JVM 2. Stream '#'.

# Created at 2024-06-05T08:41:26.098
Corrupted channel by directly writing to native stream in forked JVM 2. Stream '# JRE version: OpenJDK Runtime Environment Temurin-21.0.3+9 (21.0.3+9) (build 21.0.3+9-LTS)'.

# Created at 2024-06-05T08:41:26.098
Corrupted channel by directly writing to native stream in forked JVM 2. Stream '# Java VM: OpenJDK 64-Bit Server VM Temurin-21.0.3+9 (21.0.3+9-LTS, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-aarch64)'.

# Created at 2024-06-05T08:41:26.098
Corrupted channel by directly writing to native stream in forked JVM 2. Stream '# Problematic frame:'.

# Created at 2024-06-05T08:41:26.101
Corrupted channel by directly writing to native stream in forked JVM 2. Stream 'C  [jnilib-9883367339098451313.tmp+0x12bb0b0]  arrow::PoolBuffer::Reserve(long long)+0xc4'.

# Created at 2024-06-05T08:41:26.101
Corrupted channel by directly writing to native stream in forked JVM 2. Stream '#'.

# Created at 2024-06-05T08:41:26.102
Corrupted channel by directly writing to native stream in forked JVM 2. Stream '# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again'.

# Created at 2024-06-05T08:41:26.102
Corrupted channel by directly writing to native stream in forked JVM 2. Stream '#'.

# Created at 2024-06-05T08:41:26.104
Corrupted channel by directly writing to native stream in forked JVM 2. Stream '# An error report file with more information is saved as:'.

# Created at 2024-06-05T08:41:26.118
Corrupted channel by directly writing to native stream in forked JVM 2. Stream '# /Users/runner/work/arrow-crossbow/arrow-crossbow/arrow/java/dataset/hs_err_pid12023.log'.

# Created at 2024-06-05T08:41:26.118
Corrupted channel by directly writing to native stream in forked JVM 2. Stream '#'.

# Created at 2024-06-05T08:41:26.119
Corrupted channel by directly writing to native stream in forked JVM 2. Stream '# If you would like to submit a bug report, please visit:'.

# Created at 2024-06-05T08:41:26.121
Corrupted channel by directly writing to native stream in forked JVM 2. Stream '#   https://github.com/adoptium/adoptium-support/issues'.

# Created at 2024-06-05T08:41:26.127
Corrupted channel by directly writing to native stream in forked JVM 2. Stream '#'.

@lidavidm seems like we have an issue here

arrow::PoolBuffer::Reserve(long long)

We probably need to get this dump, download locally and debug?

lidavidm · 2024-06-05T13:18:34Z

Given the small offset I would guess that Reserve was called on a nullptr buffer

lidavidm · 2024-06-05T13:20:26Z

That said I don't think this has to block the PR?

vibhatha · 2024-06-05T14:36:10Z

Yes I think the same.

laurentgo · 2024-06-05T17:35:34Z

I was able to get a reproduction and capture jvm output, hotspot error log and core dump. I put the link to the archive in #32945 .

lidavidm · 2024-06-05T23:55:46Z

@github-actions crossbow submit -g java

github-actions · 2024-06-05T23:58:03Z

Revision: 7fbe800

Submitted crossbow builds: ursacomputing/crossbow @ actions-67dd745c51

Task	Status
java-jars
test-conda-python-3.10-spark-v3.5.0
test-conda-python-3.11-spark-master
test-conda-python-3.8-spark-v3.5.0
verify-rc-source-java-linux-almalinux-8-amd64
verify-rc-source-java-linux-conda-latest-amd64
verify-rc-source-java-linux-ubuntu-20.04-amd64
verify-rc-source-java-linux-ubuntu-22.04-amd64
verify-rc-source-java-macos-amd64

lidavidm · 2024-06-06T00:02:10Z

ci/scripts/java_full_build.sh

-find . \
-     "(" -name "*-javadoc.jar" -o -name "*-sources.jar" ")" \
-     -exec echo {} ";" \
-     -exec cp {} $dist_dir ";"


This is covered by the line below?

This is covered by the install phase which also includes the previous phases like compile or package. The same steps are already declared in the apache parent pom when using the arrow-release profile.

assembly:single was used to generate a single artifact with the whole tree and is handled by the parent plugin with the source-release-assembly execution in the parent pom (I should also remove the -DdescriptorId= property as well I guess)

source:jar is handled by attach-sources execution bound by default to package phase

javadoc:jar is handled by the attach-javadocs execution bound by default to the package phase

Ah sorry, might I misread your question. If you're asking about removing the find command, yes, it's because it's handled by the second find command which will get the single assembly, the sources and javadoc jars from the local repository and copy them to the $dist_dir directory

lidavidm · 2024-06-06T00:02:35Z

java/flight/flight-core/pom.xml

-        <!--
-          Downgrade maven-shade-plugin specifically for this module.
-          Using a newer version up to at least 3.5.1 will cause
-          issues in the arrow-tools tests looking up FlatBuffer
-          dependencies.
-         -->
-        <version>3.2.4</version>


Hmm, why does this not apply anymore? We saw problems last time we tried to upgrade

Yes, as discussed in the issue associated to the rollback, the problem was caused by the combination of arrow configuration replaced the main artifact and its associated pom with the shaded artifact and its reduced pom.xml, and the newer versions of the plugins fixing an issue where the reduced pom.xml was not attached to the artifact.

It is solved by disabling the reduced pom.xml generation and attachment using the createDependencyReducedPom configuration property

Use/update Maven modules to `org.apache:apache:31` and clean up Maven modules to remove unnecessary configuration or outdated workarounds * Add `org.apache:apache:31` to `org.apache.arrow:arrow-bom` and `org.apache.arrow.maven.plugins:arrow-maven-plugins` to make them conformant with ASF standards * Update `org.apache.arrow:arrow-java-root` parent to `org.apache:parent:31` * Use `version.*` and other properties to override plugin versions defined by `org.apache:parent` * Move standalone plugin versions under pluginManagement at the top level * Cleanup redundant plugin version or configuration declaration * Update `maven-dependency-plugin` to 3.6.1 and add the required overrides when necessary * Update `maven-shade-plugin` to 3.5.1 (via `org.apache:parent`) - disable reduced dependency pom creation for non-terminal modules * Remove enforcer check for java and maven version (handled by `org.apache:parent`) * Remove unnecessary `mvnrepository` link comments * Remove `m2e.version` property check in profiles (only needed for errorprone plugin configuration which is incompatible with M2E) * Cleanup `argLine` overrides for surefire/failsafe plugins * Remove unnecessary `../pom.xml` `<relativePath>` directives * Remove source/target/encoding configuration properties for `maven-compiler-plugin`, `maven-javadoc-plugin` and `maven-resources-plugin` as it is handled by `org.apache:parent` and plugin themselves * Remove unnecessary copy of codegen templates in `arrow-vector` module * Remove unnecessary junit jupiter engine dependencies for surefire/failsafe plugins.

Newer versions of javadoc and sources plugins do not allow attaching multiple times the "same" artifact, which causes issues with the `ci/scripts/java_full_build.sh` Update the script to remove the standalone assembly:single sources:jar and javadoc:jar which are already managed by `org.apache:apache` parent/`apache-release` profile Also remove the unnecessary copy of sources and javadoc jars from the projects as they are also installed into the local maven repository and copied from there as well Disable source-release zipfile generation in maven and bom directories as the main artifact generated by java-root will also contains both directories

lidavidm · 2024-06-07T03:29:11Z

@github-actions crossbow submit -g java

lidavidm · 2024-06-07T03:29:19Z

I'll merge once I confirm the crossbow jobs

lidavidm · 2024-06-07T03:29:30Z

Thanks @laurentgo for sticking with this!

github-actions · 2024-06-07T03:31:37Z

Revision: a7cb636

Submitted crossbow builds: ursacomputing/crossbow @ actions-a59d8b70f9

Task	Status
java-jars
test-conda-python-3.10-spark-v3.5.0
test-conda-python-3.11-spark-master
test-conda-python-3.8-spark-v3.5.0
verify-rc-source-java-linux-almalinux-8-amd64
verify-rc-source-java-linux-conda-latest-amd64
verify-rc-source-java-linux-ubuntu-20.04-amd64
verify-rc-source-java-linux-ubuntu-22.04-amd64
verify-rc-source-java-macos-amd64

There's only a single zip `arrow-java-root-{version}-source-release.zip` containnig the source directory of the whole project.

lidavidm · 2024-06-07T05:22:27Z

Ah thanks, I was just about to ask about that

lidavidm · 2024-06-07T05:22:31Z

@github-actions crossbow submit java-jars

laurentgo · 2024-06-07T05:22:58Z

I'll merge once I confirm the crossbow jobs

My bad, we discussed about the artifacts generated and I forgot to update the task verifying those. My change include a change in maven/ module to not generate a source release zip archive (containing the whole java codebase) because the main artifact arrow-java-root-{version}-source-release.zip already includes the whole tree (maven module included)

I just pushed a change to the dev task to not look for those two extra zip files (one was generated for the maven/ module and one for maven/module-info-compiler-maven-plugin)

github-actions · 2024-06-07T05:24:56Z

Revision: b64c65d

Submitted crossbow builds: ursacomputing/crossbow @ actions-d9cdcf68ef

Task	Status
java-jars

conbench-apache-arrow · 2024-06-11T14:34:39Z

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 01d2fa0.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 1 possible false positive for unstable benchmarks that are known to sometimes produce them.

laurentgo requested a review from lidavidm as a code owner May 22, 2024 05:14

github-actions bot added Component: Java awaiting review Awaiting review labels May 22, 2024

laurentgo force-pushed the laurentgo/apache-parent-update branch from a5c8d7d to 1dd63bc Compare May 23, 2024 16:04

laurentgo requested review from assignUser, kou and raulcd as code owners May 23, 2024 17:12

jbonofre approved these changes May 23, 2024

View reviewed changes

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels May 23, 2024

vibhatha mentioned this pull request May 23, 2024

[Java] java-jars fails with "We have duplicated artifacts attached" #41490

Open

laurentgo force-pushed the laurentgo/apache-parent-update branch from 37f0b16 to 55b69e3 Compare May 28, 2024 15:08

laurentgo mentioned this pull request Jun 5, 2024

[CI][Java] Sporadic failure in {{testDirectReservationListener}} #32945

Open

lidavidm approved these changes Jun 6, 2024

View reviewed changes

github-actions bot added awaiting merge Awaiting merge and removed awaiting committer review Awaiting committer review labels Jun 6, 2024

laurentgo and others added 3 commits June 6, 2024 11:41

Remove use of descriptorId property

a7cb636

laurentgo force-pushed the laurentgo/apache-parent-update branch from 0d01e78 to a7cb636 Compare June 6, 2024 09:42

Remove submodules zip files from artifact verification

b64c65d

There's only a single zip `arrow-java-root-{version}-source-release.zip` containnig the source directory of the whole project.

lidavidm merged commit 01d2fa0 into apache:main Jun 7, 2024
22 checks passed

lidavidm removed the awaiting merge Awaiting merge label Jun 7, 2024

laurentgo deleted the laurentgo/apache-parent-update branch June 7, 2024 07:19

llama90 mentioned this pull request Jun 10, 2024

[Java] Build failed: user id '1370703359' is too big ( > 2097151 ) #42068

Closed

GH-41307: [Java] Use org.apache:apache parent pom version 31 #41772

GH-41307: [Java] Use org.apache:apache parent pom version 31 #41772

Conversation

laurentgo commented May 22, 2024 • edited by github-actions bot Loading

laurentgo commented May 22, 2024

laurentgo commented May 22, 2024

github-actions bot commented May 22, 2024

laurentgo commented May 22, 2024

vibhatha commented May 22, 2024

github-actions bot commented May 22, 2024

vibhatha commented May 22, 2024

laurentgo commented May 22, 2024

vibhatha commented May 22, 2024

laurentgo commented May 23, 2024

laurentgo commented May 23, 2024

jbonofre commented May 23, 2024

vibhatha commented May 23, 2024

github-actions bot commented May 23, 2024

vibhatha commented May 23, 2024

vibhatha commented May 28, 2024

laurentgo commented May 28, 2024

vibhatha commented May 28, 2024

vibhatha commented Jun 5, 2024

vibhatha commented Jun 5, 2024

lidavidm commented Jun 5, 2024

lidavidm commented Jun 5, 2024

vibhatha commented Jun 5, 2024

laurentgo commented Jun 5, 2024

lidavidm commented Jun 5, 2024

github-actions bot commented Jun 5, 2024

lidavidm Jun 6, 2024

Choose a reason for hiding this comment

laurentgo Jun 6, 2024 • edited Loading

Choose a reason for hiding this comment

laurentgo Jun 6, 2024

Choose a reason for hiding this comment

lidavidm Jun 6, 2024

Choose a reason for hiding this comment

laurentgo Jun 6, 2024

Choose a reason for hiding this comment

lidavidm commented Jun 7, 2024

lidavidm commented Jun 7, 2024

lidavidm commented Jun 7, 2024

github-actions bot commented Jun 7, 2024

lidavidm commented Jun 7, 2024

lidavidm commented Jun 7, 2024

laurentgo commented Jun 7, 2024 • edited Loading

github-actions bot commented Jun 7, 2024

conbench-apache-arrow bot commented Jun 11, 2024

laurentgo commented May 22, 2024 •

edited by github-actions bot

Loading

laurentgo Jun 6, 2024 •

edited

Loading

laurentgo commented Jun 7, 2024 •

edited

Loading