Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Compiling with gcc 10.3 hangs/crashes with JIT when running -version #14219

Closed
pshipton opened this issue Jan 6, 2022 · 6 comments · Fixed by eclipse/omr#6301
Closed

Compiling with gcc 10.3 hangs/crashes with JIT when running -version #14219

pshipton opened this issue Jan 6, 2022 · 6 comments · Fixed by eclipse/omr#6301
Labels
comp:jit jdk17 segfault Issues that describe segfaults / JVM crashes
Milestone
Release 0.31 (Jav...

Comments

@pshipton
Copy link
Member

pshipton commented Jan 6, 2022

It's under discussion to move compilation of jdk17+ to use gcc 10.3
adoptium/temurin-build#2787

@keithc-ca tried a manual build with 10.3 and found that the build hangs in the Optimizing the exploded image step. The problem can also be seen running java -version. However running java -Xint -version completes without hanging.

@AdamBrousseau @0xdaryl fyi
@keithc-ca
Copy link
Contributor

Adding -Xint in these definitions in spec.gmk allows make images to complete.

JLINK_CMD := $(JDK_OUTPUTDIR)/bin/jlink -J-Xint
JMOD_CMD := $(JDK_OUTPUTDIR)/bin/jmod -J-Xint
BUILD_JAVA_FLAGS := -Xms64M -Xmx1600M -Xint

@keithc-ca
Copy link
Contributor

The hang seems only to occur on x86.

On POWER, the build proceeds without incident.

However, on Z it consistently crashes in the JIT:

uz16keithc1.fyre.ibm.com:

Optimizing the exploded image
#0: /home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9jit29.so(+0xa0e0fe) [0x3ffaf18e0fe]
#1: /home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9jit29.so(+0xa1b550) [0x3ffaf19b550]
#2: /home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9jit29.so(+0x16fdd4) [0x3ffae8efdd4]
#3: /home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9prt29.so(+0x3015e) [0x3ffafa3015e]
#4: [0x3ff9bff8a38]
#5: /home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9jit29.so(+0x619b6e) [0x3ffaed99b6e]
#6: /home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9jit29.so(+0xabf90c) [0x3ffaf23f90c]
#7: /home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9jit29.so(+0xac05d2) [0x3ffaf2405d2]
#8: /home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9jit29.so(+0xa9a4aa) [0x3ffaf21a4aa]
#9: /home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9jit29.so(+0xaf7414) [0x3ffaf277414]
#10: /home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9jit29.so(+0x60dc18) [0x3ffaed8dc18]
#11: /home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9jit29.so(+0x616694) [0x3ffaed96694]
#12: /home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9jit29.so(+0x6146f4) [0x3ffaed946f4]
#13: /home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9jit29.so(+0x61128e) [0x3ffaed9128e]
#14: /home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9jit29.so(+0x644d3c) [0x3ffaedc4d3c]
#15: /home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9jit29.so(+0x18adc0) [0x3ffae90adc0]
#16: /home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9jit29.so(+0x18be54) [0x3ffae90be54]
#17: /home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9prt29.so(+0x31098) [0x3ffafa31098]
#18: /home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9jit29.so(+0x189568) [0x3ffae909568]
#19: /home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9jit29.so(+0x189afc) [0x3ffae909afc]
#20: /home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9jit29.so(+0x188566) [0x3ffae908566]
#21: /home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9jit29.so(+0x188a64) [0x3ffae908a64]
#22: /home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9jit29.so(+0x188afc) [0x3ffae908afc]
#23: /home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9prt29.so(+0x31098) [0x3ffafa31098]
#24: /home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9jit29.so(+0x188f3e) [0x3ffae908f3e]
#25: /home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9thr29.so(+0x84e6) [0x3ffafc884e6]
#26: /lib/s390x-linux-gnu/libpthread.so.0(+0x9986) [0x3ffb4989986]
#27: /lib/s390x-linux-gnu/libc.so.6(+0x103cc6) [0x3ffb4c03cc6]
#28: [(nil)]
Unhandled exception
Type=Segmentation error vmState=0x0005ff06
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=000003FFAFDC2AE0 Handler2=000003FFAFA2FF40 InaccessibleAddress=0000000000000000
gpr0=000003FF00000100 gpr1=000003FF00000000 gpr2=0000000000000000 gpr3=000003FFAF57C350
gpr4=000003FFAF23F90C gpr5=0000000000000026 gpr6=000003FF00000010 gpr7=000000000000000D
gpr8=000003FF9AF04980 gpr9=000003FF9AF900D8 gpr10=000003FF9AFB4180 gpr11=0000000000000000
gpr12=000003FFAF57A0E0 gpr13=000003FFAF3EAAE8 gpr14=000003FFAED99B6E gpr15=000003FF9BFF8EC0
psw=000003FFAED99B6E mask=0705200180000000 fpc=00080000 bea=000003FFAED99980
fpr0 000003ff94007f90 (f: 2483060736.000000, d: 2.172028e-311)
fpr1 000003ffaf304810 (f: 2939176960.000000, d: 2.172254e-311)
fpr2 000003ff94008040 (f: 2483060736.000000, d: 2.172028e-311)
fpr3 48373e6900000000 (f: 0.000000, d: 7.909452e+39)
fpr4 4081800000000000 (f: 0.000000, d: 5.600000e+02)
fpr5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr6 000003ffb497c8b8 (f: 3029846272.000000, d: 2.172299e-311)
fpr7 00000000c2f29fff (f: 3270680576.000000, d: 1.615931e-314)
fpr8 000003ffac014088 (f: 2885763328.000000, d: 2.172227e-311)
fpr9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr10 000003ffac07dd40 (f: 2886196480.000000, d: 2.172228e-311)
fpr11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr12 0005d53b3b3d4b1f (f: 993872640.000000, d: 8.111695e-309)
fpr13 000002aa07da2938 (f: 131737912.000000, d: 1.447266e-311)
fpr14 000003fff547af38 (f: 4115115776.000000, d: 2.172835e-311)
fpr15 0000000000000002 (f: 2.000000, d: 9.881313e-324)
Module=/home/keithc/space/jdk17/build/normal/jdk/lib/default/libj9jit29.so
Module_base_address=000003FFAE780000

Method_being_compiled=java/lang/System.getSysPropBeforePropertiesInitialized(I)Ljava/lang/String;
Target=2_90_20220110_000000 (Linux 5.4.0-92-generic)
CPU=s390x (4 logical CPUs) (0xee36c000 RAM)
----------- Stack Backtrace -----------
(0x000003FFAED99B6E [libj9jit29.so+0x619b6e])
(0x000003FFAF23F90C [libj9jit29.so+0xabf90c])
(0x000003FFAF2405D2 [libj9jit29.so+0xac05d2])
(0x000003FFAF21A4AA [libj9jit29.so+0xa9a4aa])
(0x000003FFAF277414 [libj9jit29.so+0xaf7414])
(0x000003FFAED8DC18 [libj9jit29.so+0x60dc18])
(0x000003FFAED96694 [libj9jit29.so+0x616694])
(0x000003FFAED946F4 [libj9jit29.so+0x6146f4])
(0x000003FFAED9128E [libj9jit29.so+0x61128e])
(0x000003FFAEDC4D3C [libj9jit29.so+0x644d3c])
(0x000003FFAE90ADC0 [libj9jit29.so+0x18adc0])
(0x000003FFAE90BE54 [libj9jit29.so+0x18be54])
(0x000003FFAFA31098 [libj9prt29.so+0x31098])
(0x000003FFAE909568 [libj9jit29.so+0x189568])
(0x000003FFAE909AFC [libj9jit29.so+0x189afc])
(0x000003FFAE908566 [libj9jit29.so+0x188566])
(0x000003FFAE908A64 [libj9jit29.so+0x188a64])
(0x000003FFAE908AFC [libj9jit29.so+0x188afc])
(0x000003FFAFA31098 [libj9prt29.so+0x31098])
(0x000003FFAE908F3E [libj9jit29.so+0x188f3e])
(0x000003FFAFC884E6 [libj9thr29.so+0x84e6])
(0x000003FFB4989986 [libpthread.so.0+0x9986])
(0x000003FFB4C03CC6 [libc.so.6+0x103cc6])
(0x0000000000000000 [<unknown>+0x0])
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2022/01/10 09:34:17 - please wait.
JVMDUMP032I JVM requested System dump using '/home/keithc/space/jdk17/make/core.20220110.093417.94455.0001.dmp' in response to an event
JVMDUMP010I System dump written to /home/keithc/space/jdk17/make/core.20220110.093417.94455.0001.dmp
JVMDUMP032I JVM requested Java dump using '/home/keithc/space/jdk17/make/javacore.20220110.093417.94455.0002.txt' in response to an event
JVMDUMP010I Java dump written to /home/keithc/space/jdk17/make/javacore.20220110.093417.94455.0002.txt
JVMDUMP032I JVM requested Snap dump using '/home/keithc/space/jdk17/make/Snap.20220110.093417.94455.0003.trc' in response to an event
JVMDUMP010I Snap dump written to /home/keithc/space/jdk17/make/Snap.20220110.093417.94455.0003.trc
JVMDUMP032I JVM requested JIT dump using '/home/keithc/space/jdk17/make/jitdump.20220110.093417.94455.0004.dmp' in response to an event
JVMDUMP051I JIT dump occurred in '(unnamed thread)' thread 0x0000000000407D00
JVMDUMP049I JIT dump notified all waiting threads of the current method to be compiled
JVMDUMP054I JIT dump is tracing the IL of the method on the crashed compilation thread
JVMDUMP052I JIT dump recursive crash occurred on diagnostic thread
JVMDUMP048I JIT dump method being compiled is an ordinary method
JVMDUMP053I JIT dump is recompiling java/lang/System.getSysPropBeforePropertiesInitialized(I)Ljava/lang/String;
*** buffer overflow detected ***: terminated
JVMDUMP039I Processing dump event "abort", detail "" at 2022/01/10 09:34:19 - please wait.

@pshipton
Copy link
Member Author

vmState [0x5ff06]: {J9VMSTATE_JIT} {RegisterAssigning}

@pshipton pshipton changed the title Compiling with gcc 10.3 hangs with JIT when running -version Compiling with gcc 10.3 hangs/crashes with JIT when running -version Jan 10, 2022
@pshipton pshipton added the segfault Issues that describe segfaults / JVM crashes label Jan 10, 2022
@keithc-ca
Copy link
Contributor

More details on the crash on Z:

$ gdb jdk/bin/java
(gdb) run -version
[...snip...]
Thread 4 "JIT Compilation" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x3ffe5afe900 (LWP 98006)]
OMR::Register::getAssignedRealRegister (this=0x0) at /home/keithc/space/jdk17/omr/compiler/codegen/OMRRegister.hpp:79
79	   TR::Register *getAssignedRegister()               {return _assignedRegister;}
(gdb) bt   
#0  OMR::Register::getAssignedRealRegister (this=0x0) at /home/keithc/space/jdk17/omr/compiler/codegen/OMRRegister.hpp:79
#1  0x000003fff7c3f90c in OMR::RegisterDependencyMap::addDependency (index=0, dep=..., this=0x3ffe5af9050) at /home/keithc/space/jdk17/omr/compiler/codegen/OMRRegisterDependencyStruct.hpp:74
#2  OMR::Z::RegisterDependencyGroup::assignRegisters (this=0x3ffe4a8fd40, currentInstruction=currentInstruction@entry=0x3ffe4ab4180, kindToBeAssigned=<optimized out>, numOfDependencies=<optimized out>, 
    cg=cg@entry=0x3ffe4a038c0) at /home/keithc/space/jdk17/omr/compiler/z/codegen/OMRRegisterDependency.cpp:853
#3  0x000003fff7c405d2 in OMR::Z::RegisterDependencyConditions::assignPostConditionRegisters (this=<optimized out>, currentInstruction=currentInstruction@entry=0x3ffe4ab4180, 
    kindToBeAssigned=kindToBeAssigned@entry=TR_GPR, cg=0x3ffe4a038c0) at /home/keithc/space/jdk17/omr/compiler/z/codegen/OMRRegisterDependency.cpp:561
#4  0x000003fff7c1a4aa in TR::S390LabelInstruction::assignRegistersAndDependencies (this=0x3ffe4ab4180, kindToBeAssigned=<optimized out>) at /home/keithc/space/jdk17/omr/compiler/codegen/OMRInstruction.hpp:213
#5  0x000003fff7c77414 in OMR::Z::Instruction::assignRegisters (this=0x3ffe4ab4180, kindToBeAssigned=<optimized out>) at /home/keithc/space/jdk17/omr/compiler/codegen/OMRInstruction_inlines.hpp:30
#6  0x000003fff778dc18 in OMR::CodeGenerator::doRegisterAssignment (this=this@entry=0x3ffe4a038c0, kindsToAssign=kindsToAssign@entry=19) at /home/keithc/space/jdk17/omr/compiler/codegen/OMRCodeGenerator.cpp:876
#7  0x000003fff7796694 in OMR::CodeGenPhase::performRegisterAssigningPhase (cg=cg@entry=0x3ffe4a038c0, phase=<optimized out>) at /home/keithc/space/jdk17/omr/compiler/codegen/OMRCodeGenPhase.cpp:377
#8  0x000003fff77946f4 in OMR::CodeGenPhase::performAll (this=this@entry=0x3ffe4a03cf0) at /home/keithc/space/jdk17/omr/compiler/codegen/OMRCodeGenPhase.cpp:136
#9  0x000003fff779128e in OMR::CodeGenerator::generateCode (this=0x3ffe4a038c0) at /home/keithc/space/jdk17/omr/compiler/codegen/OMRCodeGenerator.hpp:437
#10 0x000003fff77c4d3c in OMR::Compilation::compile (this=this@entry=0x3ffe49fe000) at /home/keithc/space/jdk17/omr/compiler/compile/OMRCompilation.hpp:322
#11 0x000003fff730adc0 in TR::CompilationInfoPerThreadBase::compile (this=this@entry=0x3fff5b14070, vmThread=vmThread@entry=0x8cd00, compiler=0x3ffe49fe000, compilee=compilee@entry=0x3ffe5afc768, vm=..., 
    optimizationPlan=0x3fff5b7dd40, scratchSegmentProvider=...) at /home/keithc/space/jdk17/openj9/runtime/compiler/control/CompilationThread.cpp:9368
#12 0x000003fff730be54 in TR::CompilationInfoPerThreadBase::wrappedCompile (portLib=portLib@entry=0x3fffd2cade0 <j9portLibrary>, opaqueParameters=opaqueParameters@entry=0x3ffe5afc520)
    at /home/keithc/space/jdk17/openj9/runtime/compiler/control/CompilationThread.cpp:8871
#13 0x000003fffcc31098 in omrsig_protect (portLibrary=0x3fffd2cade0 <j9portLibrary>, fn=0x3fff730bad8 <TR::CompilationInfoPerThreadBase::wrappedCompile(J9PortLibrary*, void*)>, fn_arg=0x3ffe5afc520, 
    handler=0x3fff72efce8 <jitSignalHandler(J9PortLibrary*, uint32_t, void*, void*)>, handler_arg=0x8cd00, flags=505, result=0x3ffe5afc4f8) at /home/keithc/space/jdk17/omr/port/unix/omrsignal.c:425
#14 0x000003fff7309568 in TR::CompilationInfoPerThreadBase::compile (this=this@entry=0x3fff5b14070, vmThread=vmThread@entry=0x8cd00, entry=entry@entry=0x3fff800b7e0, scratchSegmentProvider=...)
    at /home/keithc/space/jdk17/openj9/runtime/compiler/control/CompilationThread.cpp:7882
#15 0x000003fff7309afc in TR::CompilationInfoPerThread::processEntry (this=0x3fff5b14070, entry=..., scratchSegmentProvider=...)
    at /home/keithc/space/jdk17/openj9/runtime/compiler/control/CompilationThread.cpp:4256
#16 0x000003fff7308566 in TR::CompilationInfoPerThread::processEntries (this=this@entry=0x3fff5b14070) at /home/keithc/space/jdk17/openj9/runtime/compiler/control/CompilationThread.cpp:3959
#17 0x000003fff7308a64 in TR::CompilationInfoPerThread::run (this=this@entry=0x3fff5b14070) at /home/keithc/space/jdk17/openj9/runtime/compiler/control/CompilationThread.cpp:3825
#18 0x000003fff7308afc in protectedCompilationThreadProc (compInfoPT=compInfoPT@entry=0x3fff5b14070) at /home/keithc/space/jdk17/openj9/runtime/compiler/control/CompilationThread.cpp:3759
#19 0x000003fffcc31098 in omrsig_protect (portLibrary=0x3fffd2cade0 <j9portLibrary>, fn=0x3fff7308a68 <protectedCompilationThreadProc(J9PortLibrary*, TR::CompilationInfoPerThread*)>, fn_arg=0x3fff5b14070, 
    handler=0x3fffcfc2ae0 <structuredSignalHandler>, handler_arg=0x8cd00, flags=506, result=0x3ffe5afde18) at /home/keithc/space/jdk17/omr/port/unix/omrsignal.c:425
#20 0x000003fff7308f3e in compilationThreadProc (entryarg=0x3fff5b14070) at /home/keithc/space/jdk17/openj9/runtime/compiler/control/CompilationThread.cpp:3664
#21 0x000003fffce884e6 in thread_wrapper (arg=0x3fff8007c30) at /home/keithc/space/jdk17/omr/thread/common/omrthread.c:1724
#22 0x000003fffdb09986 in start_thread (arg=0x3ffe5afe900) at pthread_create.c:477
#23 0x000003fffdd83cc6 in thread_start () at ../sysdeps/unix/sysv/linux/s390/s390-64/clone.S:65

@keithc-ca
Copy link
Contributor

I think a root cause is the definition of RegisterDependencyGroup. It defines an array of 1 RegisterDependency, but allocates space for more in the new operator.

The optimizer in gcc-10 seems to prioritize the declared array size over numberOfRegisters provided in several methods. In particular, it appears that the for loops in places like unblockRegisters() it doesn't even increment the index (no legal reference could be anything but zero) and so the loop does not terminate.

I tried making the array 'flexible' by declaring its length to be zero, only to find that the class is inherited (e.g. on X in compiler/x/codegen/OMRRegisterDependency.hpp) and extended with additional fields (a flexible array member must always be last).

My main question is this (also posted to omr#general slack channel): How did this ever work? The additional fields have offsets beginning in the same place as the second register dependency.

@keithc-ca
Copy link
Contributor

With the changes in https://github.com/keithc-ca/omr/tree/gcc10, jdk17 builds with gcc-10 on P, X and Z.

@pshipton pshipton mentioned this issue Jan 13, 2022
Closed
@keithc-ca keithc-ca mentioned this issue Jan 13, 2022
Merged
@keithc-ca keithc-ca mentioned this issue Jan 20, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
comp:jit jdk17 segfault Issues that describe segfaults / JVM crashes
Projects
None yet
Milestone
Release 0.31 (Java 18)
Development

Successfully merging a pull request may close this issue.

Fix problems compiling with gcc 10 keithc-ca/omr
2 participants
@keithc-ca @pshipton