A lot of ANRs in Google Play Console

[aliyailina]

Describe the bug:

We started using EncryptedSharedPreferences and now receive a lot of ANR from Google Play Console. There is not much information, but some of ANRs points to com.google.crypto.tink library.

What was the expected behavior?

No ANRs

How can we reproduce the bug?

Unfortunately, we have no reproduce, only data from Google Play Console.

Do you have any debugging information?

Console points to that lines of code (they are all different, it is not single stacktrace):

com.google.crypto.tink.shaded.protobuf.MessageSchema.newSchemaForRawMessageInfo+3804

com.google.crypto.tink.daead.AesSivProtoSerialization.parseKey

com.google.crypto.tink.internal.MutablePrimitiveRegistry.registerPrimitiveWrapper

com.google.crypto.tink.proto.AesSivKeyFormat.dynamicMethod

javax.crypto.Cipher.init+66

com.google.crypto.tink.shaded.protobuf.ProtobufArrayList.mutableCopyWithCapacity

com.google.crypto.tink.KeyManagerRegistry$2.getUntypedKeyManager

com.google.crypto.tink.shaded.protobuf.MessageSchema.mergeFromHelper+8504

What version of Tink are you using?

1.8.0 inside AndroidX.Security.Crypto-1.1.0-alpha06

Can you tell us more about your development environment?

Xamarin.Android
Android 24+
There is no statistically significant differences in ANR per devices or Android version.

[juergw]

This is similar to tink-crypto/tink#638. But that problem has been fixed.

But note that under the hood we are using Android Keystore, which will fail if too many threads try to access it. That might be the problem you are facing here. So what you could try is to minimize the creation of new EncryptedSharedPreference objects. For example, create the ones you need at startup, and keep reusing them. And do the same for the MasterKey object: only create one of them at startup.

I hope that helps.

[aliyailina]

@juergw Thank you for your quick response!
Yes, recreation can really be the problem. We will try to change the way we use it and see if it works.

[juergw]

Did this work?

[aliyailina]

@juergw We have no reproduce, so we need to release first and then see the results in console. I will let you know.

[aliyailina]

@juergw Unfortunately, don't think it has helped, we still see related ANRs in Console.

#00  pc 0x000000000004f55c  /apex/com.android.runtime/lib64/bionic/libc.so (syscall+28)
#01  pc 0x0000000000232858  /apex/com.android.art/lib64/libart.so (art::ConditionVariable::WaitHoldingLocks+140)
#02  pc 0x000000000045df98  /apex/com.android.art/lib64/libart.so (artJniMethodEnd+336)
#03  pc 0x00000000005c0dfc  /apex/com.android.art/lib64/libart.so (art_jni_method_end+12)
at java.lang.Thread.nativeCreate (Native method)
at java.lang.Thread.start (Thread.java:976)
at android.app.SharedPreferencesImpl.startLoadFromDisk (SharedPreferencesImpl.java:142)
at android.app.SharedPreferencesImpl.<init> (SharedPreferencesImpl.java:130)
at android.app.ContextImpl.getSharedPreferences (ContextImpl.java:577)
at android.app.ContextImpl.getSharedPreferences (ContextImpl.java:558)
at android.content.ContextWrapper.getSharedPreferences (ContextWrapper.java:217)
at com.google.crypto.tink.integration.android.AndroidKeysetManager$Builder.readKeysetFromPrefs (AndroidKeysetManager.java:251)
at com.google.crypto.tink.integration.android.AndroidKeysetManager$Builder.build (AndroidKeysetManager.java:288)
at androidx.security.crypto.EncryptedSharedPreferences.create (EncryptedSharedPreferences.java:169)
at androidx.security.crypto.EncryptedSharedPreferences.create (EncryptedSharedPreferences.java:130)
at <id>.MainApplication.n_onCreate (Native method)
at <id>.MainApplication.onCreate (MainApplication.java:26)

Now looking at stack trace I'm not even sure if it is your problem.

[juergw]

I unassigned myself because there is nothing I can do at the moment. But I'll leave this issue open for now.

[alvindizon]

In our case, our app which uses Encrypted DataStore, which in turn uses tink, have experienced ANRs from two devices, all of which are running Android 12. If you are interested, here is the stack trace:

main (timed waiting):tid=1 systid=10495 
       at java.lang.Thread.sleep(Native method)
       at java.lang.Thread.sleep(Thread.java:450)
       at java.lang.Thread.sleep(Thread.java:355)
       at android.security.KeyStoreSecurityLevel.interruptedPreservingSleep(KeyStoreSecurityLevel.java:206)
       at android.security.KeyStoreSecurityLevel.createOperation(KeyStoreSecurityLevel.java:115)
       at android.security.keystore2.AndroidKeyStoreCipherSpiBase.ensureKeystoreOperationInitialized(AndroidKeyStoreCipherSpiBase.java:334)
       at android.security.keystore2.AndroidKeyStoreCipherSpiBase.engineInit(AndroidKeyStoreCipherSpiBase.java:171)
       at javax.crypto.Cipher.tryTransformWithProvider(Cipher.java:2985)
       at javax.crypto.Cipher.tryCombinations(Cipher.java:2892)
       at javax.crypto.Cipher$SpiAndProviderUpdater.updateAndGetSpiAndProvider(Cipher.java:2797)
       at javax.crypto.Cipher.chooseProvider(Cipher.java:774)
       at javax.crypto.Cipher.init(Cipher.java:1144)
       at javax.crypto.Cipher.init(Cipher.java:1085)
       at com.google.crypto.tink.integration.android.AndroidKeystoreAesGcm.encryptInternal(AndroidKeystoreAesGcm.java:77)
       at com.google.crypto.tink.integration.android.AndroidKeystoreAesGcm.encrypt(AndroidKeystoreAesGcm.java:61)
       at com.google.crypto.tink.integration.android.AndroidKeystoreKmsClient.validateAead(AndroidKeystoreKmsClient.java:289)
       at com.google.crypto.tink.integration.android.AndroidKeystoreKmsClient.getAead(AndroidKeystoreKmsClient.java:157)
       at com.google.crypto.tink.integration.android.AndroidKeysetManager$Builder.readMasterkeyDecryptAndParseKeyset(AndroidKeysetManager.java:367)
       at com.google.crypto.tink.integration.android.AndroidKeysetManager$Builder.build(AndroidKeysetManager.java:303)
       at androidx.security.crypto.EncryptedFile$Builder.build(EncryptedFile.java:172)
       at io.github.osipxd.security.crypto.EncryptedPreferenceDataStoreSingletonDelegate$getValue$1$1.invoke(EncryptedPreferencesDataStoreDelegate.kt:121)
       at io.github.osipxd.security.crypto.EncryptedPreferenceDataStoreSingletonDelegate$getValue$1$1.invoke(EncryptedPreferencesDataStoreDelegate.kt:110)
       at io.github.osipxd.security.crypto.EncryptedPreferenceDataStoreFactoryKt.createEncrypted(EncryptedPreferenceDataStoreFactory.kt:76)
       at io.github.osipxd.security.crypto.EncryptedPreferenceDataStoreSingletonDelegate.getValue(EncryptedPreferencesDataStoreDelegate.kt:110)
       at io.github.osipxd.security.crypto.EncryptedPreferenceDataStoreSingletonDelegate.getValue(EncryptedPreferencesDataStoreDelegate.kt:83)
       at com.someapp.common.preferences.di.DataStoreModule.getEncryptedDataStore(DataStoreModule.kt:23)
       at com.someapp.common.preferences.di.DataStoreModule.provideDataStore(DataStoreModule.kt:28)
       at com.someapp.common.preferences.di.DataStoreModule_ProvideDataStoreFactory.provideDataStore(DataStoreModule_ProvideDataStoreFactory.java:50)
       at com.someapp.DaggerSomeApp_HiltComponents_SingletonC$SingletonCImpl$SwitchingProvider.get(DaggerSomeApp_HiltComponents_SingletonC.java:1320)
       at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
       at com.someapp.SomeApp_HiltComponents_SingletonC$SingletonCImpl$SwitchingProvider.get(DaggerSomeApp_HiltComponents_SingletonC.java:1317)
       at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
       at com.someapp.DaggerSomeApp_HiltComponents_SingletonC$SingletonCImpl.injectSomeApp2(DaggerSomeApp_HiltComponents_SingletonC.java:1264)
       at com.someapp.DaggerSomeApp_HiltComponents_SingletonC$SingletonCImpl.injectSomeApp(DaggerSomeApp_HiltComponents_SingletonC.java:1240)
       at com.someapp.Hilt_SomeApp.hiltInternalInject(Hilt_SomeApp.java:51)
       at com.someapp.Hilt_SomeApp.onCreate(Hilt_SomeApp.java:42)
       at com.someapp.SomeApp.onCreate(SomeApp.java:76)
       at android.app.Instrumentation.callApplicationOnCreate(Instrumentation.java:1211)
       at android.app.ActivityThread.handleBindApplication(ActivityThread.java:6765)
       at android.app.ActivityThread.access$1600(ActivityThread.java:253)
       at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2090)
       at android.os.Handler.dispatchMessage(Handler.java:106)
       at android.os.Looper.loopOnce(Looper.java:201)
       at android.os.Looper.loop(Looper.java:288)
       at android.app.ActivityThread.main(ActivityThread.java:7881)
       at java.lang.reflect.Method.invoke(Native method)
       at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:568)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1045)

AFAIK we are only invoking this once on startup, i.e., during the creation of objects via Dagger/Hilt.

[BlueTurtle3]

We have noticed the same behaviour on Android 12 devices on native Android apps using our library. In our case we are creating an EncryptedSharedPreference object with dagger once on app initialisation.

Android 12
library: androidx.security:security-crypto:1.1.0-alpha06
Native apps

[Toubap]

Same here.
Updating tink as transitive dependency might help?

[juergw]

Sorry I haven't replied earlier. Thanks for the stack-trace, that is helpful.

Tink's AndroidKeystoreAesGcm.encrypt calls Android Keystore to encrypt. But here, Android Keystore is busy, and the thread keeps retrying until it causes an ANR. This is the code where it fails:
https://android.googlesource.com/platform/frameworks/base/+/refs/heads/main/keystore/java/android/security/KeyStoreSecurityLevel.java#207

So this is not a problem in Tink. The problem is that there are too many requests to Android Keystore at the same time.

I think the problem is that dagger tries to create several EncryptFile / EncryptedSharedPreferances objects in parallel in different threads, and all threads try to decrypt the encrypted keyset at the same time. And Android keystore can't handle so many requests.

I don't know dagger. Can you make sure that these objects are created in the same thread?

[Toubap]

I do not use Dagger, and the object is created only once in a kotlin lazy property. And still experiencing the issue

[juergw]

The main problem here is that both jetpack and Tink tried to give a too simple API to the user. That API works fine in most cases, but when something goes wrong, it is much more difficult for the user of the API to figure out what is happening.

We have decided to provide a more low-level API. It requires the user to write more code themselves, but I think it makes it easier to understand what is going on, which will make it easier to debug and maintain.

Here is the new API (this will be in the next release):
https://github.com/tink-crypto/tink-java/blob/main/src/main/java/com/google/crypto/tink/integration/android/AndroidKeystore.java

and here is how a user might use this API in their app:
https://github.com/tink-crypto/tink-java/blob/main/examples/android/helloworld/app/src/main/java/com/helloworld/TinkApplication.java

[viethoangtien]

Hi @juergw, now a new version 1.1.0-alpha06 of androidx security which includes Tink Java version 1.8.0.
Does this solve this issue?

[juergw]

No, it doesn't solve the issue. Note that androidx.security:security-crypto:1.1.0-alpha06 was released on April 19, 2023, so it's not new. And it has been deprecated in the mean time:
https://developer.android.com/privacy-and-security/cryptography#jetpack_security_crypto_library