Crash in background — glFinish_Exec, gpus_ReturnNotPermittedKillClient, libGPUSupportMercury

[picciano]

I started seeing these that past two days. This was from Crashlytics, so not exactly sure what user was doing at the time.

Thread : Crashed: com.apple.main-thread
0  libGPUSupportMercury.dylib     0x000000018c389f90 gpus_ReturnNotPermittedKillClient + 12
1  libGPUSupportMercury.dylib     0x000000018c38af40 gpusSubmitDataBuffers + 172
2  GLEngine                       0x0000000187078664 glFinish_Exec + 132
3  GLKit                          0x000000018446004c -[GLKView _display:] + 200
4  QuartzCore                     0x0000000187777a34 CA::Layer::display() + 148
5  QuartzCore                     0x0000000187758a7c CA::Layer::display_if_needed(CA::Transaction*) + 244
6  QuartzCore                     0x0000000187758734 CA::Layer::layout_and_display_if_needed(CA::Transaction*) + 44
7  QuartzCore                     0x0000000187757ebc CA::Context::commit_transaction(CA::Transaction*) + 276
8  QuartzCore                     0x0000000187757c3c CA::Transaction::commit() + 528
9  UIKit                          0x0000000187e0c56c _afterCACommitHandler + 152
10 CoreFoundation                 0x00000001833902a4 __CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ + 32
11 CoreFoundation                 0x000000018338d230 __CFRunLoopDoObservers + 360
12 CoreFoundation                 0x000000018338d610 __CFRunLoopRun + 836
13 CoreFoundation                 0x00000001832b92d4 CFRunLoopRunSpecific + 396
14 GraphicsServices               0x000000018cacf6fc GSEventRunModal + 168
15 UIKit                          0x0000000187e7efac UIApplicationMain + 1488
16 TallyGo                        0x00000001001284f0 main (main.m:14)
17 libdyld.dylib                  0x000000019527aa08 start + 4

[friedbunny]

This looks suspiciously similar to the two crashes discussed in #1198, but not quite. @picciano, do you see any hardware/OS patterns for these crashes?

[picciano]

I've been seeing it on iPhone 6, iOS versions 8.2 and 8.3. This build is using the MBGL codebase from Monday, May 4, late afternoon.

[friedbunny]

It looks like gpus_ReturnNotPermittedKillClient usually means that the app is trying to render OpenGL in the background, says Apple and variety of Stack Overflow threads (like this one).

Here is effectively the same crash on VLC's iOS app. It appears they fixed it by checking for a broader range of background notifications.

[friedbunny]

@incanus just hardened iOS backgrounding behavior yesterday in #1451. We don't handle UIApplicationWillTerminateNotification, but I'm not sure we need to?

[incanus]

I don't think we need to hit UIApplicationWillTerminateNotification.

We are still seeing this crash internally, though not in a way that looks like a crash to anyone (because of the state-saving nature of backgrounding). Currently trying to work out what exactly the repro is.

[incanus]

http://stackoverflow.com/a/15814933/977220

I now keep track of the state of the status bar in relation to the app's active status, and only animate if the application state is UIApplicationStateActive. Having done this, I haven't had a crash since.

[picciano]

According to Crashlytics, the app was not in background. But perhaps it's just a reporting/timing glitch?

[incanus]

Hmm, @picciano, any way you could share some of the log around that, or explain how you know?

[picciano]

See screen shot image pasted in above message. Crashlytics is reporting 100% of the 8 crashes were with "app in focus".

[1ec5]

Does Crashlytics rely on the same UIApplicationDidEnterBackgroundNotification/UIApplicationWillEnterForegroundNotification notifications we’ve been using? If so, it’d be misreporting exactly these crashes as being “while in focus”.

[incanus]

We just can't seem to repro this internally. We are "seeing" the crash in Hockey, but no one is reporting any crash problems and we can't repro in Xcode. Maybe we can punt on this?

[picciano]

wouldn't worry about it for beta 1 at least, but am seeing it from two different users/phones

[incanus]

Yeah, same here.

[incanus]

Ok, moving this to b2, but we can move back if we see it concretely and/or get more details.

Marking as "ignored" for now in Hockey.

[incanus]

Does Crashlytics rely on the same

Yeah, couldn't find any docs on this, so it's hard to tell what they mean. App state? No modals? Has been gestured-on since foregrounding? etc.

[picciano]

I did find a way to recreate this reliably. Launch app, put app into background, make a change in the app's settings. (In my case, changing permission to access Contacts.) Boom goes the crash.

[incanus]

Thanks @picciano, I'll check this out. Could be related to metrics in the Settings.app while backgrounding the app.

[incanus]

Not seeing this in my own app settings testing, neither for metrics nor location services. To clarify, @picciano you are seeing a GL-related crash when changing Contacts access permissions? Do you respond to that change in any way in your app? Particularly in a GUI way? I know this would be odd in the background, but I'm trying to figure out what about settings would affect the GL view, or the view drawing at all.

[picciano]

This is still mysterious...I am as confused as you.

1. Yes, I am seeing this GL-related crash when changing contacts access permission.
2. I do not respond to that change in the app. In fact, the view controller that uses those settings doesn't even need to have been loaded.
3. I just recently added the contacts permission to the app and the other users experienced the crash before contacts permission request was added to the app. I am also asking the user for permission to access location at all times, not just while in foreground. I only added this method as an alternative way to generate the same error as it seems to be related to app settings changes.

The only good thing about this is that it crashes while in the Settings app, so the average user doesn't even really notice. (okay, still bad)

[1ec5]

As an experiment, I rigged the iOS demo app to ask for access to contacts. I ran the app in the debugger and granted access. As soon as I went into Settings and switched off that access, the app was sent SIGKILL. -[GLKView _display:] was never called:

Thread 1Queue : com.apple.main-thread (serial)
#0  0x0000000195d3fbd0 in objc_msgSend ()
#1  0x0000000184538b7c in __CFBasicHashAddValue ()
#2  0x00000001843e4684 in CFDictionarySetValue ()
#3  0x0000000185384b08 in _encodeObject ()
#4  0x000000018538c35c in +[NSKeyedArchiver archivedDataWithRootObject:] ()
#5  0x000000018a9e4724 in +[NSObject(BaseBoard) bs_dataFromObject:] ()
#6  0x000000018a9ad57c in BKSHIDSetEventFocusDeferrals ()
#7  0x000000018a9a9a94 in -[BKSEventFocusManager reallyFlushWithSet:] ()
#8  0x000000018a9a971c in -[BKSEventFocusManager flush] ()
#9  0x0000000100b8cfd4 in _dispatch_call_block_and_release ()
#10 0x0000000100b8cf94 in _dispatch_client_callout ()
#11 0x0000000100b91c28 in _dispatch_main_queue_callback_4CF ()
…

[kkaefer]

the app was sent SIGKILL.

This might be by design?

[1ec5]

Yes, it's definitely by design, to prevent apps from hanging onto contact information longer than the user allows. If this also happens when not running in the debugger, then it's unlikely to be the case that the crash happens when toggling the setting via Settings, with the app in the background.

[incanus]

So we've proven a link between merely changing the contact settings and the app getting killed, which alludes to @picciano's problem but isn't necessarily the direct cause. I wonder @picciano, are you doing anything else related to app lifecycle such as hiding or snapshotting views that would be triggered by this?

[picciano]

No. It's a pretty typical navigation controller-style app. I am using the same instance of map view on a couple different views, but nothing weird.

And to be clear, I started seeing this crash pop up in Crashlytics from our testers before I was able to reproduce it myself. In fact, I was seeing in Crashlytics before the Contact permission request was even in the app. The only other app setting is the permissions to access the user location at all times, not just when the app is foreground and the metrics privacy settings.

[picciano]

I am still seeing this in the latest build. The speculative fix did not work.

[1ec5]

@picciano, yes, #1805 clearly did not work. Can you try #1821? The fix there needs to be made more resilient against calls to change the style URL etc., but I think it at least addresses the root cause.

[picciano]

@iec5 i'll drop that into tonight's build and let you know

[picciano]

After bringing in the changes from #1821 I saw this in Crashlytics.

Thread : Fatal Exception: NSRangeException
0  CoreFoundation                 0x22220137 __exceptionPreprocess
1  libobjc.A.dylib                0x302b0c77 objc_exception_throw
2  CoreFoundation                 0x2222007d -[NSException initWithCoder:]
3  Foundation                     0x22ebccc9 -[NSObject(NSKeyValueObserverRegistration) _removeObserver:forProperty:]
4  Foundation                     0x22ebc6db -[NSObject(NSKeyValueObserverRegistration) removeObserver:forKeyPath:]
5  TallyGo                        0x00328bf3 -[MGLMapView dealloc] (MGLMapView.mm:438)
6  TallyGo                        0x00326787 -[MGLMapView initWithFrame:styleURL:] (MGLMapView.mm:138)
7  TallyGo                        0x0010d5af -[TGMapUtility init] (TGMapUtility.m:143)
8  TallyGo                        0x0010cc9d __30+[TGMapUtility sharedInstance]_block_invoke (TGMapUtility.m:24)
9  libdispatch.dylib              0x3081ac6f _dispatch_client_callout
10 libdispatch.dylib              0x3082908f dispatch_once_f$VARIANT$mp
11 TallyGo                        0x0010cc69 +[TGMapUtility sharedInstance] (once.h:68)
12 TallyGo                        0x0015c66b -[TGMainViewController viewDidLoad] (TGMainViewController.m:109)
13 UIKit                          0x2585a7ad -[UIViewController loadViewIfRequired]
14 UIKit                          0x25905545 -[UINavigationController _layoutViewController:]
15 UIKit                          0x2590546d -[UINavigationController _updateScrollViewFromViewController:toViewController:]
16 UIKit                          0x259049e1 -[UINavigationController _startTransition:fromViewController:toViewController:]
17 UIKit                          0x25904717 -[UINavigationController _startDeferredTransitionIfNeeded:]
18 UIKit                          0x25904485 -[UINavigationController __viewWillLayoutSubviews]
19 UIKit                          0x25904419 -[UILayoutContainerView layoutSubviews]
20 UIKit                          0x25857c1b -[UIView(CALayerDelegate) layoutSublayersOfLayer:]
21 QuartzCore                     0x25276445 -[CALayer layoutSublayers]
22 QuartzCore                     0x25271c95 CA::Layer::layout_if_needed(CA::Transaction*)
23 QuartzCore                     0x25271b1d CA::Layer::layout_and_display_if_needed(CA::Transaction*)
24 QuartzCore                     0x252714bf CA::Context::commit_transaction(CA::Transaction*)
25 QuartzCore                     0x252712af CA::Transaction::commit()
26 UIKit                          0x25abe1f5 -[UIApplication _reportMainSceneUpdateFinished:]
27 UIKit                          0x25abef7d -[UIApplication _runWithMainScene:transitionContext:completion:]
28 UIKit                          0x25ac9809 __84-[UIApplication _handleApplicationActivationWithScene:transitionContext:completion:]_block_invoke
29 UIKit                          0x25abd2e7 -[UIApplication workspaceDidEndTransaction:]
30 FrontBoardServices             0x28ce0ec1 __31-[FBSSerialQueue performAsync:]_block_invoke_2
31 CoreFoundation                 0x221e6255 __CFRUNLOOP_IS_CALLING_OUT_TO_A_BLOCK__
32 CoreFoundation                 0x221e5519 __CFRunLoopDoBlocks
33 CoreFoundation                 0x221e4073 __CFRunLoopRun
34 CoreFoundation                 0x2212f9a1 CFRunLoopRunSpecific
35 CoreFoundation                 0x2212f7b3 CFRunLoopRunInMode
36 UIKit                          0x258bfab7 -[UIApplication _run]
37 UIKit                          0x258ba695 UIApplicationMain
38 TallyGo                        0x0015bb83 main (main.m:14)
39 libdyld.dylib                  0x30858aaf start

[1ec5]

Looks like we're getting somewhere, being able to detect background launches now. Should be easy to fix the crash above: there's no need to unregister for KVO registrations before the registrations take place (registration is deferred in this case).

@picciano, let's continue the conversation in #1821, since that looks promising.

[picciano]

Sorry, meant #1821 (not 1812)

[incanus]

@1ec5 is heading this up.

[1ec5]

Reopening because the fix was backed out in 2df80af due to #1855. Hopefully we can bring back the fix, sans regression, in a minor update.