CRIU TR::CompilationInfo::prepareForCheckpoint causes hang intermittently at single thread mode #15191
Comments
|
@dsouzai please take a look at this |
|
@JasonFengJ9 do you mind sending me the diagnostic info (and the jdk)? |
|
@dsouzai pls find attached test.zip It is an intermittent problem, ~ |
|
I was able to reproduce a hang, but I don't see a thread waiting in I have a couple of questions:
As it stands, it doesn't look like it's the CR Monitor to blame, but I'll try to reproduce some more to see. |
In this mode, only the thread performing Checkpoint is expected to run,
It was from a live process attached, I can try reproducing and get the stacktraces of the JIT threads. |
|
Annoyingly enough, I can't reproduce this when I build the JVM locally (even with the same OMR, OpenJ9, and extensions repo SHAs). I think I do see one problem in the JIT that can manifest itself as the thread waiting on the CR Monitor. If there are comp threads that are "active" but waiting on work, they wait on the Comp Monitor. When the hook thread tries to suspend all the comp threads, it doesn't send a notify on the Comp Monitor, so it's possible for it to wait forever while the "active" comp threads sit waiting to be notified to compile something (which never happens because no java code is running anymore). I'll open a PR to fix that, which I believe is the problem seen in the stack trace in this issue. However, there might be a second issue because in my env I don't ever see anyone waiting on the CR Monitor. I do see that one of the comp threads is waiting to be notified on the comp monitor, but it doesn't look like we even get to the jit hook where it would hang waiting on the CR monitor. |
|
Got lucky and managed to get a hang with my local build which contains the fix for the thread waiting on the comp monitor. It doesn't look we ever got to the JIT hook. I'm now more convinced there are two problems; I don't think the second problem is caused by the JIT. |
|
Opened #15202 to fix the JIT issue. |
The testcase intends to create monitor contention leading to a hang which is expected to be discovered by #14651, and throw |
|
Ah I see, so should I just have #15202 close this issue automatically? |
Yes, pls do. |
|
Based on #15191 (comment) I decided to pull in the changes from #14651. With those changes, I'm able to reproduce the original problem. While the issue of the comp thread waiting on the comp monitor is, I believe, valid, the actual problem here seems to be because a comp thread is compiling, and is blocked waiting for exclusive vm access: I'm not really sure what a good way forward is. We do want to have the ability to compile before performing a checkpoint but I guess @tajila @JasonFengJ9 do you guys have any suggestions? |
|
Do you think it's safe for the thread that's hooking into the jit to release vmaccess while it waits on the various monitors, and then reacquire it right before we return from the hook (assuming it's the hook thread that has vmaccess)? |
|
With this change: I didn't see the issue even after running 500 times. I still have a couple of questions though:
|
Single Thread Mode stopped all threads that can run Java code, and GC/JIT threads are not affected. So yes, it is possible other threads holding the vmaccess, DDR command
Does |
No, it's the opposite. In the code in #15191 (comment) I'm releasing vmaccess in
Will they hold on to it indefinitely, or will they release it at some point, because otherwise the compilation thread will block indefinitely waiting for exclusive vmaccess.
I was asking is it ok for the hook thread (the thread that calls |
Just to clarify the question about
That depends on what other GC/JIT threads are doing, the compilation thread could be blocked during STM.
Assuming |
|
So here's what the circumstance looks like: Comp Thread (C1): compiling some method. Hook Thread (T1): VM calls openj9/runtime/compiler/control/HookedByTheJit.cpp Lines 1832 to 1841 in 6335127 T1 may have vmaccess when it's in C1 needs exclusive vmaccess at some point during the compilation. However, if any other threads have vmaccess, it's going to have to wait. When in C1 is not going to suspend itself until it's done compiling. However, it can't finish compiling until it acquires exclusive vmaccess. The simple solution I put up is that if T1 has vmaccess, it releases it before waiting for all comp threads to suspend themselves, and then reacquires it before returning back to the VM. The thing I'm not sure about is, is that good enough? The constraint here is, we want to have the ability to compile a bunch of methods before we checkpoint, and as such, we need to allow comp threads to acquire exclusive vmaccess while T1 is waiting in Is my solution of having T1 release vmaccess at the start of |
Yes, prepareForCheckpoint is always called with VMAccess. It should be safe to release VMAccess at the start of jitHookPrepareCheckpoint and reaquire again at the end because at this point we have halted all java threads. |
|
Is this only required for |
Yeah this is only needed on the checkpoint side; on the restore side all we do is resume the comp threads to go back to looking for work. We don't allow a checkpoint while a compilation is in flight. |
While working on a testcase for Prevent contended monitor enters at CRIU single thread mode, it hangs intermittently, and the culprit thread has following stacktrace:
fyi @dsouzai @tajila
The text was updated successfully, but these errors were encountered: