OpenJDK java/lang/StringBuilder/HugeCapacity Unexpected exit from test [exit code: 137]

[pshipton]

This was seen once before, #17203 (comment)

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/529 - ub20-390-4
jdk_lang_j9_0
-Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage -Xjit -Xgcpolicy:balanced -Xnocompressedrefs
java/lang/StringBuilder/HugeCapacity.java

I don't spot any diagnostics.

00:36:55  ACTION: main -- Failed. Unexpected exit from test [exit code: 137]
00:36:55  REASON: User specified action: run main/othervm -Xmx5G -XX:+CompactStrings HugeCapacity 
00:36:55  TIME:   17.562 seconds
00:36:55  messages:
00:36:55  command: main -Xmx5G -XX:+CompactStrings HugeCapacity
00:36:55  reason: User specified action: run main/othervm -Xmx5G -XX:+CompactStrings HugeCapacity 
00:36:55  Mode: othervm [/othervm specified]
00:36:55  elapsed time (seconds): 17.562
00:36:55  configuration:
00:36:55  STDOUT:
00:36:55  STDERR:
00:36:55  rerun:
00:36:55  cd /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/TKG/output_16829975804315/jdk_lang_j9_0/work/scratch/0 && \
00:36:55  DISPLAY=:0 \
00:36:55  HOME=/home/jenkins \
00:36:55  LANG=en_US.UTF-8 \
00:36:55  PATH=/bin:/usr/bin:/usr/sbin \
00:36:55  CLASSPATH=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/TKG/output_16829975804315/jdk_lang_j9_0/work/classes/1/java/lang/StringBuilder/HugeCapacity.d:/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/StringBuilder:/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/jvmtest/openjdk/jtreg/lib/javatest.jar:/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/jvmtest/openjdk/jtreg/lib/jtreg.jar \
00:36:55      /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/openjdkbinary/j2sdk-image/bin/java \
00:36:55          -Dtest.vm.opts='-ea -esa -Xmx512m -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage -Xjit -Xgcpolicy:balanced -Xnocompressedrefs' \
00:36:55          -Dtest.tool.vm.opts='-J-ea -J-esa -J-Xmx512m -J-Xdump:system:none -J-Xdump:heap:none -J-Xdump:system:events=gpf+abort+traceassert+corruptcache -J-XX:-JITServerTechPreviewMessage -J-Xjit -J-Xgcpolicy:balanced -J-Xnocompressedrefs' \
00:36:55          -Dtest.compiler.opts= \
00:36:55          -Dtest.java.opts= \
00:36:55          -Dtest.jdk=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/openjdkbinary/j2sdk-image \
00:36:55          -Dcompile.jdk=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/openjdkbinary/j2sdk-image \
00:36:55          -Dtest.timeout.factor=8.0 \
00:36:55          -Dtest.nativepath=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/openjdkbinary/openjdk-test-image/jdk/jtreg/native \
00:36:55          -Dtest.root=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk \
00:36:55          -Dtest.name=java/lang/StringBuilder/HugeCapacity.java \
00:36:55          -Dtest.file=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/StringBuilder/HugeCapacity.java \
00:36:55          -Dtest.src=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/StringBuilder \
00:36:55          -Dtest.src.path=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/StringBuilder \
00:36:55          -Dtest.classes=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/TKG/output_16829975804315/jdk_lang_j9_0/work/classes/1/java/lang/StringBuilder/HugeCapacity.d \
00:36:55          -Dtest.class.path=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/TKG/output_16829975804315/jdk_lang_j9_0/work/classes/1/java/lang/StringBuilder/HugeCapacity.d \
00:36:55          -Dtest.class.path.prefix=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/TKG/output_16829975804315/jdk_lang_j9_0/work/classes/1/java/lang/StringBuilder/HugeCapacity.d:/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/StringBuilder \
00:36:55          -ea \
00:36:55          -esa \
00:36:55          -Xmx512m \
00:36:55          -Xdump:system:none \
00:36:55          -Xdump:heap:none \
00:36:55          -Xdump:system:events=gpf+abort+traceassert+corruptcache \
00:36:55          -XX:-JITServerTechPreviewMessage \
00:36:55          -Xjit \
00:36:55          -Xgcpolicy:balanced \
00:36:55          -Xnocompressedrefs \
00:36:55          -Djava.library.path=/home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/openjdkbinary/openjdk-test-image/jdk/jtreg/native \
00:36:55          -Xmx5G \
00:36:55          -XX:+CompactStrings \
00:36:55          com.sun.javatest.regtest.agent.MainWrapper /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/TKG/output_16829975804315/jdk_lang_j9_0/work/java/lang/StringBuilder/HugeCapacity.d/main.0.jta
00:36:55  
00:36:55  TEST RESULT: Failed. Unexpected exit from test [exit code: 137]
00:36:55  --------------------------------------------------
00:40:13  Test results: passed: 731; failed: 1

[pshipton]

40x https://openj9-jenkins.osuosl.org/job/Grinder/2307/ - passed

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.openjdk_s390x_linux_OpenJDK11/68 - ub20-390-6

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.openjdk_s390x_linux_OMR/4 - ub20-390-5
https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/536 - ub20-390-3

---

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Nightly/537 - ub20-390-6

[pshipton]

@dmitripivkine not sure if this belongs to GC. The failure started showing up when we starting testing with -Xgcpolicy:balanced not long ago. We've only seen it on zlinux. It's only seen on jdk11 because the test is excluded from jdk16 (17) due to #10582

[dmitripivkine]

@pshipton @llxia Is there any way to run test differently to get failure details to look at? I can not find any results stored, just error message. There is no reason to run grinders until we figure out how to get meaningful results.

[pshipton]

Checking if the failure can be reproduced when running just the failing test.
Using EXTRA_OPTIONS -Xgcpolicy:balanced -Xnocompressedrefs -Xdump:system:events=systhrow,filter=java/lang/OutOfMemoryError
50x x 5 grinder https://openj9-jenkins.osuosl.org/job/Grinder/2476/ - passed on ub20-390-1, 3, 4, 5, 6
50x x 5 grinder https://openj9-jenkins.osuosl.org/job/Grinder/2477/ - passed on ub20-390-3, 5, 6, rh7-390-1, 4

[pshipton]

Try running all jdk_lang_j9_0
Using EXTRA_OPTIONS -Xdump:system:events=systhrow,filter=java/lang/OutOfMemoryError, which I expect will create a number of core files for passing tests.
3x x 5 https://openj9-jenkins.osuosl.org/job/Grinder/2478/

[pshipton]

There is a test mismatch in the grinder, but HugeCapacity also failed in 2/5 of the runs (runs 0, 4 on ub20-390-3, 4). The other runs occurred on the rhel7 machines.

The test does only seem to fail on the ub20 machines. It seems like maybe there isn't enough memory to create the JVM and run the test. All the machines have about 8GB, but the ub20 machines seem to be configured with no swap space, while the
rhel7 machines do have swap space. I'll set this as an infra issue.

[JasonFengJ9]

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Release_testList_1/5/consoleFull

04:35:04  TEST RESULT: Failed. Unexpected exit from test [exit code: 137]
04:35:04  --------------------------------------------------
04:37:39  Test results: passed: 730; failed: 1
04:37:55  Report written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Release_testList_1/aqa-tests/TKG/output_16900990055977/jdk_lang_j9_0/report/html/report.html
04:37:55  Results written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Release_testList_1/aqa-tests/TKG/output_16900990055977/jdk_lang_j9_0/work
04:37:55  Error: Some tests failed or other problems occurred.
04:37:55  -----------------------------------
04:37:55  jdk_lang_j9_0_FAILED

[pshipton]

Also fails on rh8-390-2, which also has no swap space. Since we aren't making any progress fixing the machines I will exclude the test until it can pass.

              total        used        free      shared  buff/cache   available
Mem:        7696108     1538712     5004988      368900     1152408     5672452
Swap:             0           0           0

adoptium/aqa-tests#4940

[pshipton]

The machines are updated to have swap space, and seem to work. Unexclude the test.
adoptium/aqa-tests#5033