Kernel boot stuck at idle after optee. Need help.

[ZhikuiRen]

I am porting optee into a new ArmA7 chip with the flow similar to OP-TEE boot flow #1887.
When optee-os switch to normal mode, Linux kernel stuck at idle state and kernel_init thread fails to run.
If I remove the optee-os, the same kernel build would boot correctly.
What can the problem be? I appreciate any suggestions how to identify the issue.

##Here is the log from the qemu seesion:

Loading loadables from FIT Image at 00080000 ...
Trying 'kernel@1' loadables subimage
Description: Linux kernel
Type: Kernel Image
Compression: uncompressed
Data Start: 0x00080114
Data Size: 5712096 Bytes = 5.4 MiB
Architecture: ARM
OS: Linux
Load Address: 0x80001000
Entry Point: 0x80001000
Verifying Hash Integrity ... OK
Loading loadables from 0x00080114 to 0x80001000 <===this is linux kernel
Loading Kernel Image ... OK <=== this is optee-os
Loading Ramdisk to 8ee39000, end 90000000 ... OK
using: FDT
Loading Device Tree to 8ee23000, end 8ee381b2 ... OK <===this device tree for kernel
Transferring control to Linux (at address bb000000)...

Starting kernel ...

D/TC:0 0 console_init:100 Early console on UART
D/TC:0 0 add_phys_mem:575 VCORE_UNPG_RX_PA type TEE_RAM_RX 0xbb000000 size 0x0004f000
D/TC:0 0 add_phys_mem:575 VCORE_UNPG_RW_PA type TEE_RAM_RW 0xbb04f000 size 0x000b1000
D/TC:0 0 add_phys_mem:575 TA_RAM_START type TA_RAM 0xbb100000 size 0x02d00000
D/TC:0 0 add_phys_mem:575 TEE_SHMEM_START type NSEC_SHM 0xbde00000 size 0x00100000
D/TC:0 0 add_phys_mem:575 CONSOLE_UART_BASE type IO_NSEC 0x1e784000 size 0x00100000
D/TC:0 0 add_phys_mem:575 ROUNDDOWN((0x40461000 + 0x1000), CORE_MMU_PGDIR_SIZE) type IO_SEC 0x40400000 size 0x00100000
D/TC:0 0 add_phys_mem:575 ROUNDDOWN((0x40461000 + 0x2000), CORE_MMU_PGDIR_SIZE) type IO_SEC 0x40400000 size 0x00100000
D/TC:0 0 add_phys_mem:589 Physical mem map overlaps 0x40400000
D/TC:0 0 verify_special_mem_areas:514 No NSEC DDR memory area defined
D/TC:0 0 add_va_space:615 type RES_VASPACE size 0x00a00000
D/TC:0 0 add_va_space:615 type SHM_VASPACE size 0x02000000
D/TC:0 0 dump_mmap_table:746 type IO_NSEC va 0xb5300000..0xb53fffff pa 0x1e784000..0x1e883fff size 0x00100000 (smallpg)
D/TC:0 0 dump_mmap_table:746 type TA_RAM va 0xb5400000..0xb80fffff pa 0xbb100000..0xbddfffff size 0x02d00000 (pgdir)
D/TC:0 0 dump_mmap_table:746 type NSEC_SHM va 0xb8200000..0xb82fffff pa 0xbde00000..0xbdefffff size 0x00100000 (pgdir)
D/TC:0 0 dump_mmap_table:746 type SHM_VASPACE va 0xb8300000..0xba2fffff pa 0x00000000..0x01ffffff size 0x02000000 (pgdir)
D/TC:0 0 dump_mmap_table:746 type RES_VASPACE va 0xba400000..0xbadfffff pa 0x00000000..0x009fffff size 0x00a00000 (pgdir)
D/TC:0 0 dump_mmap_table:746 type IO_SEC va 0xbaf00000..0xbaffffff pa 0x40400000..0x404fffff size 0x00100000 (pgdir)
D/TC:0 0 dump_mmap_table:746 type TEE_RAM_RX va 0xbb000000..0xbb04efff pa 0xbb000000..0xbb04efff size 0x0004f000 (smallpg)
D/TC:0 0 dump_mmap_table:746 type TEE_RAM_RW va 0xbb04f000..0xbb0fffff pa 0xbb04f000..0xbb0fffff size 0x000b1000 (smallpg)
D/TC:0 0 core_mmu_alloc_l2:272 L2 table used: 1/4
D/TC:0 0 core_mmu_alloc_l2:272 L2 table used: 2/4
I/TC:
D/TC:0 0 init_canaries:161 #Stack canaries for stack_tmp[0] with top at 0xbb06e9b8
D/TC:0 0 init_canaries:161 watch *0xbb06e9bc
D/TC:0 0 init_canaries:161 #Stack canaries for stack_tmp[1] with top at 0xbb06f2f8
D/TC:0 0 init_canaries:161 watch *0xbb06f2fc
D/TC:0 0 init_canaries:162 #Stack canaries for stack_abt[0] with top at 0xbb06fb38
D/TC:0 0 init_canaries:162 watch *0xbb06fb3c
D/TC:0 0 init_canaries:162 #Stack canaries for stack_abt[1] with top at 0xbb070378
D/TC:0 0 init_canaries:162 watch *0xbb07037c
D/TC:0 0 init_canaries:164 #Stack canaries for stack_thread[0] with top at 0xbb0723b8
D/TC:0 0 init_canaries:164 watch *0xbb0723bc
I/TC: OP-TEE version: 3.7.0-3-g74b3a23-dev (gcc version 9.2.0 (GCC)) #33 Mon Apr 13 04:28:20 UTC 2020 arm
D/TC:0 0 check_ta_store:638 TA store: "Secure Storage TA"
D/TC:0 0 check_ta_store:638 TA store: "REE"
D/TC:0 0 mobj_mapped_shm_init:446 Shared memory address range: b8300000, ba300000
I/TC: Initialized
D/TC:0 0 init_primary_helper:1109 Primary CPU switching to normal world boot

gdb shows where the kernel is hung:
(gdb) c
Continuing.
^C
Thread 1 received signal SIGINT, Interrupt.
cpu_v7_do_idle () at /usr/src/kernel/arch/arm/mm/proc-v7.S:78
78 ret lr
(gdb) info threads
Id Target Id Frame

• 1 Thread 1.1 (CPU#0 [halted ]) cpu_v7_do_idle () at /usr/src/kernel/arch/arm/mm/proc-v7.S:78
  2 Thread 1.2 (CPU#1 [halted ]) cpu_v7_do_idle () at /usr/src/kernel/arch/arm/mm/proc-v7.S:78
  (gdb) bt
  #0 cpu_v7_do_idle () at /usr/src/kernel/arch/arm/mm/proc-v7.S:78
  Update Notice.md #1 0x8010a08c in arch_cpu_idle () at /usr/src/kernel/arch/arm/kernel/process.c:71
  Update aeabi_divmod.c #2 0x808aad50 in default_idle_call () at /usr/src/kernel/kernel/sched/idle.c:94
  Add constant time memcmp_ct function #3 0x80153894 in cpuidle_idle_call () at /usr/src/kernel/kernel/sched/idle.c:154
  Kta cleanup #4 do_idle () at /usr/src/kernel/kernel/sched/idle.c:263
  GitHub usage documentation #5 0x80153bf8 in cpu_startup_entry (state=CPUHP_ONLINE) at /usr/src/kernel/kernel/sched/idle.c:355
  add README.md #6 0x808a30d8 in rest_init () at /usr/src/kernel/init/main.c:451
  Update aeabi_divmod.c #7 0x80b00d1c in arch_call_rest_init () at /usr/src/kernel/init/main.c:573
  Fix warning in sta_helloworld #8 0x80b011d0 in start_kernel () at /usr/src/kernel/init/main.c:785
  Add THREAD_FLAGS_EXIT_ON_IRQ for preempted threads #9 0x00000000 in ?? ()

[jenswi-linaro]

Perhaps interrupt controller isn't configured correctly (no timer interrupts)?
Incorrect/unexpected register content (r0-r3 or so) on kernel entry?
The kernel log is empty, is this because we hang before console has been initialized?

[ZhikuiRen]

I also suspect the interrupt is not working. But I don't know where to look to pin point the issue. I used gdb stepping through the code, but that does not help me to see why the interrupt is not firing. Or maybe, optee-os trapped those. What is the best way to find out?

The code actually finishes "console_init" and I can see all the printk messages in the log buffer, but I don't know why it does not show up after console_init. Can kernel share the same UART/console that optee-os uses? ( There is only one physical serial port in our board.)

[github-actions]

This issue has been marked as a stale issue because it has been open (more than) 30 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 5 days. Note, that you can always re-open a closed issue at any time.

[ZhikuiRen]

The issue is resolved by registering GIC registers with MEM_AREA_IO_SEC in the platform code, i.e. register_phys_mem(MEM_AREA_IO_SEC, GICD_BASE, size). QEMU emulate secure access based on the memory attributes, not the NSR bit. Closeing this issue.