Debug build Hangs during Startup

[ingorichter]

Original Title: Brackets debug build on OSX crashes during startup

• open the appshell.xcodeproject
• create a debug build
• launch debug build

Result: the app crashes.

Same result on Windows.

[gruehle]

Low priority to @ingorichter

[redmunds]

I needed the Mac debugger last week and ran into this. It doesn't seem like "low priority" when you need it. :)

Using git bisect, I isolated this problem to commit adobe/brackets-shell@fdea13a from pull request adobe/brackets-shell#281 by @RaymondLim.

The code seems to go into an infinite loop. The menu works, but nothing happens when you select anything.

Re-assigned to @RaymondLim .

[redmunds]

I cannot run a debug build on Windows either. Seems similar -- caught in an infinite loop. Looks like maybe a resource deadlock?

Here's the callstack:

    ntdll.dll!_ZwWaitForMultipleObjects@20()  + 0x15 bytes  
    ntdll.dll!_ZwWaitForMultipleObjects@20()  + 0x15 bytes  
    kernel32.dll!_WaitForMultipleObjectsExImplementation@20()  + 0x8e bytes 
    user32.dll!_RealMsgWaitForMultipleObjectsEx@20()  + 0xe2 bytes  
    libcef.dll!1013ad01()   
    [Frames below may be incorrect and/or missing, no symbols loaded for libcef.dll]    
    libcef.dll!1013aa64()   
    libcef.dll!10139de2()   
    libcef.dll!1013a09c()   
    libcef.dll!10119956()   
    libcef.dll!1011972e()   
    libcef.dll!10144619()   
    libcef.dll!10118f2b()   
    libcef.dll!1022bd96()   
    libcef.dll!1019e3ea()   
    libcef.dll!1000cd58()   
>   Brackets.exe!CefRunMessageLoop()  Line 252 + 0x8 bytes  C++
    Brackets.exe!wWinMain(HINSTANCE__ * hInstance=0x002e0000, HINSTANCE__ * hPrevInstance=0x00000000, wchar_t * lpCmdLine=0x006a2888, int nCmdShow=1)  Line 253 C++
    Brackets.exe!__tmainCRTStartup()  Line 275 + 0x2c bytes C
    Brackets.exe!wWinMainCRTStartup()  Line 189 C
    kernel32.dll!@BaseThreadInitThunk@12()  + 0x12 bytes    
    ntdll.dll!___RtlUserThreadStart@8()  + 0x27 bytes   
    ntdll.dll!__RtlUserThreadStart@8()  + 0x1b bytes    

[redmunds]

Bumping this up to Medium Priority.

[RaymondLim]

@redmunds I'm seeing the same thing on Mac (editor window becomes blank like WSOD) even after I revert my changes in adobe/brackets-shell#281.

Below is the call stack on Mac and I think it's more useful than the one @redmunds got on Windows with explicit debugger break. It's a segmentation fault and maybe we should reach out to CEF team.

ASSERTION FAILED: foundContainer
../../third_party/WebKit/Source/core/rendering/RenderGeometryMap.cpp(99) : void WebCore::RenderGeometryMap::mapToContainer(WebCore::TransformState &, const WebCore::RenderLayerModelObject ) const
1 0x17b5fe3 WebCore::RenderGeometryMap::mapToContainer(WebCore::TransformState&, WebCore::RenderLayerModelObject const) const
2 0x17b6537 WebCore::RenderGeometryMap::mapToContainer(WebCore::FloatRect const&, WebCore::RenderLayerModelObject const_) const
3 0x16fbb9f WebCore::RenderBox::outlineBoundsForRepaint(WebCore::RenderLayerModelObject const_, WebCore::RenderGeometryMap const_) const
4 0x17d7cf1 WebCore::RenderLayer::computeRepaintRects(WebCore::RenderLayerModelObject const_, WebCore::RenderGeometryMap const_)
5 0x17d9d4f WebCore::RenderLayer::updateLayerPositionsAfterScroll(WebCore::RenderGeometryMap_, unsigned int)
6 0x17d9f68 WebCore::RenderLayer::updateLayerPositionsAfterOverflowScroll()
7 0x17e1330 WebCore::RenderLayer::setScrollOffset(WebCore::IntPoint const&)
8 0x3b04d52 WebCore::ScrollableArea::scrollPositionChanged(WebCore::IntPoint const&)
9 0x3b05271 WebCore::ScrollableArea::setScrollOffsetFromAnimation(WebCore::IntPoint const&)
10 0x3af7ed0 WebCore::ScrollAnimator::notifyPositionChanged(WebCore::FloatSize const&)
11 0x3e21680 WebCore::ScrollAnimatorMac::notifyPositionChanged(WebCore::FloatSize const&)
12 0x3e213e5 WebCore::ScrollAnimatorMac::immediateScrollTo(WebCore::FloatPoint const&)
13 0x3e21254 WebCore::ScrollAnimatorMac::scrollToOffsetWithoutAnimation(WebCore::FloatPoint const&)
14 0x3b04a00 WebCore::ScrollableArea::scrollToOffsetWithoutAnimation(WebCore::FloatPoint const&)
15 0x17e0c92 WebCore::RenderLayer::scrollToOffset(WebCore::IntSize const&, WebCore::RenderLayer::ScrollOffsetClamping)
16 0x17e1e68 WebCore::RenderLayer::scrollRectToVisible(WebCore::LayoutRect const&, WebCore::ScrollAlignment const&, WebCore::ScrollAlignment const&)
17 0x18990f9 WebCore::RenderObject::scrollRectToVisible(WebCore::LayoutRect const&, WebCore::ScrollAlignment const&, WebCore::ScrollAlignment const&)
18 0x431f8b3 WebCore::FrameSelection::revealSelection(WebCore::ScrollAlignment const&, WebCore::RevealExtentOption)
19 0x2542e86 WebCore::HTMLTextAreaElement::updateFocusAppearance(bool)
20 0x4b7fdc1 WebCore::Element::focus(bool, WebCore::FocusDirection)
21 0x56fc8a3 WebCore::ElementV8Internal::focusMethod(v8::FunctionCallbackInfov8::Value const&)
22 0x56fa78b WebCore::ElementV8Internal::focusMethodCallback(v8::FunctionCallbackInfov8::Value const&)
23 0x3677e2 v8::internal::FunctionCallbackArguments::Call(v8::Handlev8::Value ()(v8::Arguments const&))
24 0x3b3bad v8::internal::MaybeObject v8::internal::HandleApiCallHelper(v8::internal::(anonymous namespace)::BuiltinArguments<(v8::internal::BuiltinExtraArguments)1>, v8::internal::Isolate_)
25 0x3b369b v8::internal::Builtin_Impl_HandleApiCall(v8::internal::(anonymous namespace)::BuiltinArguments<(v8::internal::BuiltinExtraArguments)1>, v8::internal::Isolate_)
26 0x3a830a v8::internal::Builtin_HandleApiCall(int, v8::internal::Object*, v8::internal::Isolate)
27 0x24a0a576
28 0x4a094744
Received signal 11 SEGV_MAPERR 0000bbadbeef
[0x00000204a86f]
[0x00000204a80b]
[0x00000204a49d]
[0x000099abe94b]
[0x0000ffffffff]
[0x0000017b6537]
[0x0000016fbb9f]
[0x0000017d7cf1]
[0x0000017d9d4f]
[0x0000017d9f68]
[0x0000017e1330]
[0x000003b04d52]
[0x000003b05271]
[0x000003af7ed0]
[0x000003e21680]
[0x000003e213e5]
[0x000003e21254]
[0x000003b04a00]
[0x0000017e0c92]
[0x0000017e1e68]
[0x0000018990f9]
[0x00000431f8b3]
[0x000002542e86]
[0x000004b7fdc1]
[0x0000056fc8a3]
[0x0000056fa78b]
[0x0000003677e2]
[0x0000003b3bad]
[0x0000003b369b]
[0x0000003a830a]
[0x000024a0a576]
[0x00004a094744]
ax: bbadbeef, bx: bfff8100, cx: 73dbbfe0, dx: 73dbbfe0
di: 7beb32a, si: 7beb2b0, bp: bfff7f98, sp: bfff7f00, ss: 23, flags: 10282
ip: 17b5fed, cs: 1b, ds: 23, es: 23, fs: 0, gs: f

[RaymondLim]

Pull request adobe/brackets-shell#355 only fixes the specific issue on mac and still does not fix the other pre-existing issue that causes Brackets window going blank.

[RaymondLim]

@redmunds
As I already mentioned above, my pull request adobe/brackets-shell#281 is not the real culprit of this debug crash issue. I can understand why your use of "git bisect" identified it as the culprit. It does cause the debug hang happen earlier. However, if you revert my changes in that pull request, you will still get the debug crash issue (WSOD). Besides, my pull request is to fix a crash on Mac and should not affect the Windows build but you're having this debug crash issue on Windows as well.

If you review the calll stack I pasted above, you will see that the crash (segmentation fault) is happening when rendering the editor. So I began suspecting that the crash was introduced in Brackets repo and not in Brackets-shell repo. And It took me several tries on brackets repo using "git bisect" and finally I found out that it is caused by @njx's commit 119b4ef for inline editor animation. I did verify that the debug crash happens with that commit (on both Windows and Mac platforms), but no crash if I checkout the prior one 73ca611.

So reassigning to @njx to investigate it. Maybe it is a valid bug in CEF that we just uncover with this specific commit, but it will be nice if we can make some changes in our code to avoid this debug issue.

[peterflynn]

Interesting... @njx I wonder if it's the webkit-backface-visibility? Most of the other changes in that commit should only kick in when an inline editor is actually opened, which won't be the case right at startup time. Whereas webkit-backface-visibility applies to the whole body, and does something unspecified with the GPU acceleration pipeline... which is always a little sketchy :-)

[RaymondLim]

@peterflynn
You're absolutely right! I commented out both webkit-backface-visibility and backface-visibility and my debug build no longer crashes.

Oops... spoke too soon. I just got a crash after switching project, but definitely not that easy to crash w/o these CSS properties. Below is the call stack I got with this change.

ASSERTION FAILED: m_verifier.isSafeToUse()
../../third_party/WebKit/Source/wtf/RefCounted.h(59) : void WTF::RefCountedBase::ref()
1 0x1969617 WTF::RefCountedBase::ref()
2 0x47710cd void WTF::refIfNotNullWebCore::UserGestureToken(WebCore::UserGestureToken_)
3 0x475efe8 WTF::RefPtrWebCore::UserGestureToken::operator=(WebCore::UserGestureToken_)
4 0x471daa9 WebCore::DOMTimer::DOMTimer(WebCore::ScriptExecutionContext_, WTF::PassOwnPtrWebCore::ScheduledAction, int, bool)
5 0x471d995 WebCore::DOMTimer::DOMTimer(WebCore::ScriptExecutionContext_, WTF::PassOwnPtrWebCore::ScheduledAction, int, bool)
6 0x471df14 WebCore::DOMTimer::install(WebCore::ScriptExecutionContext_, WTF::PassOwnPtrWebCore::ScheduledAction, int, bool)
7 0x6027f80 WebCore::SetTimeoutOrInterval(v8::FunctionCallbackInfov8::Value const&, bool)
8 0x602845c WebCore::V8WorkerContext::setTimeoutMethodCustom(v8::FunctionCallbackInfov8::Value const&)
9 0x5b307bb WebCore::WorkerContextV8Internal::setTimeoutMethodCallback(v8::FunctionCallbackInfov8::Value const&)
10 0x3677e2 v8::internal::FunctionCallbackArguments::Call(v8::Handlev8::Value ()(v8::Arguments const&))
11 0x3b3bad v8::internal::MaybeObject* v8::internal::HandleApiCallHelper(v8::internal::(anonymous namespace)::BuiltinArguments<(v8::internal::BuiltinExtraArguments)1>, v8::internal::Isolate)
12 0x3b369b v8::internal::Builtin_Impl_HandleApiCall(v8::internal::(anonymous namespace)::BuiltinArguments<(v8::internal::BuiltinExtraArguments)1>, v8::internal::Isolate_)
13 0x3a830a v8::internal::Builtin_HandleApiCall(int, v8::internal::Object*, v8::internal::Isolate)
14 0x5470a576
Received signal 11 SEGV_MAPERR 0000bbadbeef
[0x00000204a86f]
[0x00000204a80b]
[0x00000204a49d]
[0x000099abe94b]
[0x0000ffffffff]
[0x0000047710cd]
[0x00000475efe8]
[0x00000471daa9]
[0x00000471d995]
[0x00000471df14]
[0x000006027f80]
[0x00000602845c]
[0x000005b307bb]
[0x0000003677e2]
[0x0000003b3bad]
[0x0000003b369b]
[0x0000003a830a]
[0x00005470a576]
ax: bbadbeef, bx: 16d96a01, cx: a96e4752, dx: a96e4752
di: 7c1d667, si: 7c1d7c7, bp: b007f958, sp: b007f920, ss: 23, flags: 10286
ip: 1969621, cs: 1b, ds: 23, es: 23, fs: 23, gs: f

[njx]

That second crash looks like a v8 crash, so it might not be related to the renderer.

[redmunds]

Yay! I can run Debug builds on both Mac and Windows again!

[RaymondLim]

But you still can't change project with the dropdown. Otherwise, you will hit the second crash.

[redmunds]

I didn't test exhaustively, but I was able to switch to another project and then switch back to the first project.

[RaymondLim]

Ok, the second crash doesn't happen on every click. But occasionally it can happen when hovering, clicking and double-clicking on file/folder in the file tree or X button before file in working set or the project dropdown list.

[njx]

Not sure I'm the right person to look at this since I don't have much experience debugging in the shell. @JeffryBooher or @bchintx maybe?

[JeffryBooher]

I've not have any problems on OSX debugging although it is slower.

The work-around that I suggest is to use the release binaries of CEF for building the debug version of the shell. That should get you past all of the assertions that seem to come up in the debug builds but crashers in the renderer may still occur in the release build of CEF. If it is a reproducible case then it's best to get that datapoint to Stefan-Teodor Craciun [email protected] and see if he can figure it out.

Kind of hard to follow this thread but it sounds like this particular instance of the crash is in V8, so it will require someone with security access to V8 source in order to debug it or at least enough knowledge to know how to get it reported correctly to the V8 team.

I'd start with running with the release binaries. If you can get past your crash with the release binaries then I don't think there is anything else we need to do about it.

[bchintx]

+1 to @JeffryBooher 's comment. I'm not seeing a crash debugging the shell on Windows now either, and commenting out the and in the .less files does seem to help the debugging slowly trudge thru the startup.

In addition to his workaround, on Windows, I'd also suggest just sandwiching any code that you want to debug with the following optimization-disabling switches:

#pragma optimize( "", off ) #pragma warning( disable: 4748 ) . // the code you want to step thru here . #pragma optimize( "", on )

Now, you can continue to build Release, be able to step thru the code quickly, but have non-mangled, full debug symbols for that section of code.

[ingorichter]

@bchintx @JeffryBooher @redmunds Did anybody see this recently? Should we close it for now?

[redmunds]

@ingorichter I haven't needed a Debug build in while, but I can try again.

[JeffryBooher]

@ingorichter the Windows Debug build still crashes on Startup even with CEF 1750. It's something in our JS that's causing it. It's the render process that dies.

[nethip]

I think it is the node connection that is causing this crash in debug builds. Commenting startNodeProcess in cefclient_win.cpp is taking away the crash. We are yet to figure out which part of the node connection is causing an assert. The thing with CEF helper is that, it just crashes on asserts, which is a pain. I will update this thread as and when we have more information on this.

CC @marcelgerber @redmunds @vadhawal