after long time playback, tee encounters core data-abort

[dhclouisyang]

Dear All,
We integrate the TEE OS in our android platform (a53, quad-core) and test with the long time DRM playback testing. It encounters the tee core crash and the log as below.
We think the problem is due to tee core data-abort, so we can see the linux kernel stall.
Can anyone give me the hints or advices why the tee core data-abort ?
We have no idea how to deal with the issue. Thanks !!

<kernel crash log>
08-03 17:15:47.235 0 0 E : INFO: rcu_preempt detected stalls on CPUs/tasks:
08-03 17:15:47.235 0 0 E : 3: (1 GPs behind) idle=3f1/140000000000000/0 softirq=122478651/122478652 fqs=3149
08-03 17:15:47.235 0 0 E : (detected by 0, t=6302 jiffies, g=61101404, c=61101403, q=300)
08-03 17:15:47.235 0 0 I : Task dump for CPU 3:
08-03 17:15:47.235 0 0 I Binder : 2655_10 R running task 0 23155 1 0x0040000a
08-03 17:15:47.236 0 0 F Call trace:
08-03 17:15:47.236 0 0 W : [] __switch_to+0x8c/0xa4
08-03 17:15:47.236 0 0 W : [] __schedule+0x1b0/0x65c
08-03 17:15:47.236 0 0 W : [] preempt_schedule_irq+0x3c/0x90
08-03 17:15:47.236 0 0 W : [] el1_preempt+0x8/0x50
08-03 17:15:47.236 0 0 W : [] optee_invoke_func+0x128/0x188
08-03 17:15:47.236 0 0 W : [] tee_ioctl+0xa38/0xc2c
08-03 17:15:47.236 0 0 W : [] compat_SyS_ioctl+0xbc/0xfd8

<TEE crash log>
[Thu Aug 03 17:15:22.159 2017] ERROR: TEE-CORE:
[Thu Aug 03 17:15:22.159 2017] ERROR: TEE-CORE: core data-abort at address 0xffffffffffffffb1 (alignment fault)
[Thu Aug 03 17:15:22.165 2017] ERROR: TEE-CORE: esr 0x96000061 ttbr0 0x1000010526360 ttbr1 0x00000000 cidr 0x0
[Thu Aug 03 17:15:22.172 2017] ERROR: TEE-CORE: cpu #3 cpsr 0x20000104
[Thu Aug 03 17:15:22.178 2017] ERROR: TEE-CORE: x0 ffffffffffffffb1 x1 0000000000000015
[Thu Aug 03 17:15:22.184 2017] ERROR: TEE-CORE: x2 0000000000000004 x3 0000000000000004
[Thu Aug 03 17:15:22.191 2017] ERROR: TEE-CORE: x4 0000000000000007 x5 0000000010520018
[Thu Aug 03 17:15:22.198 2017] ERROR: TEE-CORE: x6 0000000000000000 x7 0000000000000000
[Thu Aug 03 17:15:22.204 2017] ERROR: TEE-CORE: x8 0000000000000049 x9 000000001051fd80
[Thu Aug 03 17:15:22.210 2017] ERROR: TEE-CORE: x10 0000000000000000 x11 00000000000000c0
[Thu Aug 03 17:15:22.216 2017] ERROR: TEE-CORE: x12 0000000000000000 x13 00000000401007b0
[Thu Aug 03 17:15:22.222 2017] ERROR: TEE-CORE: x14 0000000000000010 x15 0000000000000000
[Thu Aug 03 17:15:22.229 2017] ERROR: TEE-CORE: x16 000000001021f83c x17 0000000000000000
[Thu Aug 03 17:15:22.236 2017] ERROR: TEE-CORE: x18 0000000000000000 x19 0000000098015874
[Thu Aug 03 17:15:22.242 2017] ERROR: TEE-CORE: x20 0000000000000000 x21 000000001028b500
[Thu Aug 03 17:15:22.248 2017] ERROR: TEE-CORE: x22 00000000000005c0 x23 0000000000000000
[Thu Aug 03 17:15:22.254 2017] ERROR: TEE-CORE: x24 0000000000000000 x25 0000000000000000
[Thu Aug 03 17:15:22.261 2017] ERROR: TEE-CORE: x26 0000000000000000 x27 0000000000000000
[Thu Aug 03 17:15:22.267 2017] ERROR: TEE-CORE: x28 0000000000000000 x29 000000001051ff10
[Thu Aug 03 17:15:22.273 2017] ERROR: TEE-CORE: x30 0000000010203e3c elr 0000000010203b18
[Thu Aug 03 17:15:22.280 2017] ERROR: TEE-CORE: sp_el0 000000001051fed0
[Thu Aug 03 17:15:22.280 2017] ERROR: TEE-CORE: [abort] alignement fault! (trap CPU)
[Thu Aug 03 17:15:22.286 2017] ERROR: TEE-CORE: PANIC: get_fault_type core/arch/arm/kernel/abort.c:513

[jforissier]

Hi @dhclouisyang,

"Core data-abort" means the CPU has tried to access an invalid virtual address from EL1. That address is 0xffffffffffffffb1.
Which version of OP-TEE are you using? With 2.5.0 you should have a stack strace which would probably be quite helpful to troubleshoot the issue.
Still, you have the ELR register, that should help find the location of the fault. Try:
aarch64-linux-gnu-addr2line -f -p -e out/*/core/tee.elf 0000000010203b18

Your analysis of the Linux error message is correct. When an unrecoverable error such as a core data-abort occurs in OP-TEE, the CPU core is put into a spin loop so it won't break anything (or won't trigger the same fault again and again). The consequence is, the CPU core appears to be stuck to Linux and you get a "rcu_preempt dtected stalls" message.

[dhclouisyang]

Hi Jforissier,
We use OP-TEE 2.1 released version.
Is it possible to merge the v2.5 stack strace code to our current OP-TEE 2.1 code base ??

Thanks !!

[jforissier]

Hi @dhclouisyang,

Cherry-pick commit a681fab ("trace.h: add macros to unwind and print the call stack (kernel only)") and build with CFG_CORE_UNWIND=y. Then you should obtain a call stack on abort.

Some later commits introduce minor reformatting of the stack trace as well as a script to convert addresses to function name/file name/line number, but they depend on the introduction of user TA stack dump (#1552), so they will be harder to cherry-pick.

[github-actions]

This issue has been marked as a stale issue because it has been open (more than) 30 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 5 days. Note, that you can always re-open a closed issue at any time.