imx7d/imx6d rom_start section relocation for proper remoteproc firmware loading

[jlh-makeen]

@DerekSnell @JiafeiPan

To follow our discussion on Discord :-)

I would like to discuss the fact that IMX7D and IMX6 need the bootvector aka sp/pc to be located into the OCRAM_S area for proper booting using remoteproc.

As of now, Zephyr places the isr vectors (the rom_start section) into ocram/ocram_s/tcm/ddr as per the chosen zephyr,flash parameter in the DT.

But the linux rproc driver doesn't have the code necessary to systematically copy the first 8 bytes of the rom_start section onto the boot_vector for the M4 as mentionned by NXP's AN5317.

Instead the elf-loader is supposed to load the section properly as per the definitions in the linker file.

When you look at FREERTOS examples the full vector table itself is placed explicitely into OCRAM_S regardless of is the code itself is located in OCRAM/DDR/TCM.
See here:
https://github.com/toradex/FreeRTOS-Colibri-iMX7/blob/master/platform/devices/MCIMX7D/linker/gcc/MCIMX7D_M4_ocram.ld

It actually means that for all rproc that needs SP/PC to be loaded onto a specific location, we need to relocate the the rom_start section onto a specific memory area, a change of the linker script is necessary.
This should be a IMX7D or IMX6 global configuration as this issue will occur on all boards' M cores running zephyr.

At the moment, I've implemented it in my zephyr/fork:

I've done the following changes:

Adding a new memory area in the linker script with kconfig flags:

• CONFIG_ROMSTART_CUSTOM_REGION : activate the custom rom_start section generation
• CONFIG_ROMSTART_REGION_ADDRESS : base address of the rom_start section
• CONFIG_ROMSTART_REGION_SIZE : size in KB of the rom_start section

in /zephyr/include/arch/arm/cortex_m/scripts/linker.ld

[...]

/* physical address of RAM */
#ifdef CONFIG_XIP
#define ROMABLE_REGION FLASH
#else
#define ROMABLE_REGION RAM
#endif
#define RAMABLE_REGION RAM

/* Some SOCs like IMX6 and IMX7 M4 need the IRQ vector table to be placed at a special address see AN5317 from NXP */
/* other socs simply place it at the start of ROMABLE_REGION */
#if defined(CONFIG_ROMSTART_CUSTOM_REGION)
#define ROMSTART_ADDR CONFIG_ROMSTART_REGION_ADDRESS
#define ROMSTART_SIZE (CONFIG_ROMSTART_REGION_SIZE * 1K)
#else
#define ROMSTART_REGION ROMABLE_REGION
#endif

#if !defined(CONFIG_XIP) && (CONFIG_FLASH_SIZE == 0)
#define ROM_ADDR RAM_ADDR
#else
#define ROM_ADDR (CONFIG_FLASH_BASE_ADDRESS + CONFIG_FLASH_LOAD_OFFSET)
#endif

[...]

MEMORY
    {
#if defined(CONFIG_ROMSTART_CUSTOM_REGION)
    ROMSTART_REGION (rx) : ORIGIN = ROMSTART_ADDR, LENGTH = ROMSTART_SIZE
#endif
    FLASH (rx) : ORIGIN = ROM_ADDR, LENGTH = ROM_SIZE
    RAM   (wx) : ORIGIN = RAM_ADDR, LENGTH = RAM_SIZE
#if defined(CONFIG_LINKER_DEVNULL_MEMORY)
    DEVNULL_ROM (rx) : ORIGIN = DEVNULL_ADDR, LENGTH = DEVNULL_SIZE
#endif
    LINKER_DT_REGIONS()
    /* Used by and documented in include/linker/intlist.ld */
    IDT_LIST (wx) : ORIGIN = 0xFFFF7FFF, LENGTH = 32K
    }

ENTRY(CONFIG_KERNEL_ENTRY)

SECTIONS
    {

#include <zephyr/linker/rel-sections.ld>

    /*
     * .plt and .iplt are here according to 'arm-zephyr-elf-ld --verbose',
     * before text section.
     */
    /DISCARD/ :
	{
	*(.plt)
	}

    /DISCARD/ :
	{
	*(.iplt)
	}

    GROUP_START(ROMABLE_REGION)

	__rom_region_start = ROM_ADDR;

    SECTION_PROLOGUE(rom_start,,)
	{

/* Located in generated directory. This file is populated by calling
 * zephyr_linker_sources(ROM_START ...). This typically contains the vector
 * table and debug information.
 */
#include <snippets-rom-start.ld>

	} GROUP_LINK_IN(ROMSTART_REGION)

#ifdef CONFIG_CODE_DATA_RELOCATION

#include <linker_relocate.ld>

#endif /* CONFIG_CODE_DATA_RELOCATION */

and then in /zephyr/soc/nxp/imx/imx7d/Kconfig.defconfig:

# iMX7 core series

# Copyright 2017,2024 NXP
# SPDX-License-Identifier: Apache-2.0

if SOC_SERIES_IMX7D

config NUM_IRQS
	# must be >= the highest interrupt number used
	default 127

config PINCTRL_IMX
	default y if HAS_IMX_IOMUXC
	depends on PINCTRL

config ROMSTART_CUSTOM_REGION
	bool "Custom region definition to place the rom_start boot vector data"
	default y

if ROMSTART_CUSTOM_REGION

	#OCRAM_S is the boot vector for IMX7D/M4 at 0x00180000 with 32kB size
	config ROMSTART_REGION_ADDRESS
		hex "Start address of irq vector rom_start region"
		default $(dt_nodelabel_reg_addr_hex,ocram_s_sys)
		
	config ROMSTART_REGION_SIZE
		hex "Size of the irq vector rom_start region in kB"
		default $(dt_nodelabel_reg_size_hex,ocram_s_sys,0,K)

endif

rsource "Kconfig.defconfig.*"

endif # SOC_SERIES_IMX7D

In turns that generates the correct elf file with the section for rom_start always being located into the OCRAM_S 0x000180000 location.
In the following app, my code is placed in OCRAM (base 0x00900000).

readelf -S build/zephyr/zephyr.elf 
There are 43 section headers, starting at offset 0x21e5c4:

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] rom_start         PROGBITS        00180000 000114 00023c 00 WAX  0   0  4
  [ 2] text              PROGBITS        00900000 000350 01232c 00  AX  0   0  8
  [ 3] .ARM.exidx        ARM_EXIDX       0091232c 01267c 000008 00  AL  2   0  4
  [ 4] initlevel         PROGBITS        00912334 012684 0000a8 00   A  0   0  4
  [ 5] device_area       PROGBITS        009123dc 01272c 0000f0 00   A  0   0  4
  [ 6] sw_isr_table      PROGBITS        009124cc 01281c 0003f8 00  WA  0   0  4
  [ 7] ctors             PROGBITS        009128c4 012c14 00000c 00  WA  0   0  1

Remoteproc now boots the M4 correctly with these changes :-) even if it wasn't started from u-boot.

Any comments and ideas if this is a good idea to upstream this change and if yes, how would you recommend proceeding ?

[JiafeiPan]

I am glad to see you refine my suggestions in Discord into a more general implementation, I think it is okay to upstream these updates but with some extra suggestions:

1. Is it better to use CONFIG_ROMSTART_RELOCATION_ROM but not CONFIG_ROMSTART_CUSTOM_REGION?
2. Move CONFIG_ROMSTART_RELOCATION_ROM to be a common configuration item in zephyr/arch/arm/Kconfig, so that maybe can be leveraged by other platforms.
3. A more detailed help information for CONFIG_ROMSTART_RELOCATION_ROM to introduce when this item is needed
4. I think the address and size of relocation ROM can be defined by dts nodes, and for imx7d, 1K bytes is enough.

Maybe you can initial an PR, then have more discussion with Maintainers.

[jlh-makeen]

Thanks for your suggestions @JiafeiPan, Relocation is probably a better wording here I agree with your points.
I'll prepare a PR as soon as I have progressed more on my application side :)

I need to cleanup my zephyr branch to make up for a clean PR commit.

One more reflexion though, how does it come that no one has mentioned this boot-vector issue on IMX before ?

I mean how does it come that we have all the RPMSG/mbox driver and openamp stacks implemented properly and no one noticed the issue of remoteproc not loading the boot-vector of the M4 core ?

How could have people tested/developped the rpmsg stack if their M-core wasn't even started properly by rproc ?
Does everyone just "bootaux" in u-boot and trust it to load the bootvector ? then only use the start/stop commands of the rproc sysfs to trigger the RPMSG NS annoucements and mount the tty's ?

[JiafeiPan]

I think currently OCRAM_S is used as ROM memory by default, so there is no such issue.

[jlh-makeen]

Well yeah that could have been an explaination:

But pretty much all of the imx7d boards I've checked use tcm memory so I'm still wondering how this subject didn't come up before 🤔

Nevermind, I'll provide a PR as this is the right way to solve the issue.
Especially because these boards are used in the ipm test samples:

zephyr/samples/drivers/ipm/ipm_imx/sample.yaml

Line 12 in 10fafa0

- warp7/mcimx7d/m4

https://github.com/zephyrproject-rtos/zephyr/blob/main/boards/toradex/colibri_imx7d/colibri_imx7d_mcimx7d_m4.dts

https://github.com/zephyrproject-rtos/zephyr/blob/main/boards/technexion/pico_pi/pico_pi_mcimx7d_m4.dts

https://github.com/zephyrproject-rtos/zephyr/blob/10fafa025ea199182776bf35113762a7ef67d4fc/boards/element14/warp7/warp7_mcimx7d_m4.dts

https://github.com/zephyrproject-rtos/zephyr/blob/10fafa025ea199182776bf35113762a7ef67d4fc/boards/96boards/meerkat96/96b_meerkat96_mcimx7d_m4.dts

[neuberfran]

@jlh-makeen Pls include pico_pi_m4 in Your PR zephyr/samples/drivers/ipm/ipm_imx/sample.yaml

[jlh-makeen]

The way I've implemented it at the moment is that this will be a default config for all boards using mcimx7d/m4, therefore automatically supporting pico_m4 too.

[jlh-makeen]

@JiafeiPan

While I was preparing the PR, I figured something out in the linux imx_rproc #driver:
https://github.com/STMicroelectronics/linux/blob/v6.1-stm32mp/drivers/remoteproc/imx_rproc.c

The address translation tables indicate that the address 0 is translated tot the right boot-vector location.
It is not the case for all IMXs, but for imx7d, imx8m and imx6sx it is translated correctly that address 0 points to OCRAM_S for IMX7D, TCM_L for imx8m and TCM_L for imx6sx.

Therefore a default value of 0x00000000 and size 1K for the bootvector by default KCONFIG is correct for IMX7D/IMX6SX/IMX8M.

static const struct imx_rproc_att imx_rproc_att_imx7d[] = {
	/* dev addr , sys addr  , size	    , flags */
	/* OCRAM_S (M4 Boot code) - alias */
	{ 0x00000000, 0x00180000, 0x00008000, 0 },
	/* OCRAM_S (Code) */
	{ 0x00180000, 0x00180000, 0x00008000, ATT_OWN },
	/* OCRAM (Code) - alias */
	{ 0x00900000, 0x00900000, 0x00020000, 0 },
	/* OCRAM_EPDC (Code) - alias */
	{ 0x00920000, 0x00920000, 0x00020000, 0 },
	/* OCRAM_PXP (Code) - alias */
	{ 0x00940000, 0x00940000, 0x00008000, 0 },
	/* TCML (Code) */
	{ 0x1FFF8000, 0x007F8000, 0x00008000, ATT_OWN | ATT_IOMEM },
	/* DDR (Code) - alias, first part of DDR (Data) */
	{ 0x10000000, 0x80000000, 0x0FFF0000, 0 },

	/* TCMU (Data) */
	{ 0x20000000, 0x00800000, 0x00008000, ATT_OWN | ATT_IOMEM },
	/* OCRAM (Data) */
	{ 0x20200000, 0x00900000, 0x00020000, 0 },
	/* OCRAM_EPDC (Data) */
	{ 0x20220000, 0x00920000, 0x00020000, 0 },
	/* OCRAM_PXP (Data) */
	{ 0x20240000, 0x00940000, 0x00008000, 0 },
	/* DDR (Data) */
	{ 0x80000000, 0x80000000, 0x60000000, 0 },
};

static const struct imx_rproc_att imx_rproc_att_imx6sx[] = {
	/* dev addr , sys addr  , size	    , flags */
	/* TCML (M4 Boot Code) - alias */
	{ 0x00000000, 0x007F8000, 0x00008000, ATT_IOMEM },
	/* OCRAM_S (Code) */
	{ 0x00180000, 0x008F8000, 0x00004000, 0 },
	/* OCRAM_S (Code) - alias */
	{ 0x00180000, 0x008FC000, 0x00004000, 0 },
	/* TCML (Code) */
	{ 0x1FFF8000, 0x007F8000, 0x00008000, ATT_OWN | ATT_IOMEM },
	/* DDR (Code) - alias, first part of DDR (Data) */
	{ 0x10000000, 0x80000000, 0x0FFF8000, 0 },

	/* TCMU (Data) */
	{ 0x20000000, 0x00800000, 0x00008000, ATT_OWN | ATT_IOMEM },
	/* OCRAM_S (Data) - alias? */
	{ 0x208F8000, 0x008F8000, 0x00004000, 0 },
	/* DDR (Data) */
	{ 0x80000000, 0x80000000, 0x60000000, 0 },
};

I also need to have a default flag for CONFIG_ROMSTART_RELOCATION_ROM selected only if the selected zephyr,flash reg address doesn't match the boot-vector address.

I have tested it on my IMX7D and it still boots when placing the code in OCRAM and boot-vector at address 0x00000000 into 1K size

readelf -S build/zephyr/zephyr.elf 
There are 43 section headers, starting at offset 0x21e5c4:

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] rom_start         PROGBITS        00000000 000114 00023c 00 WAX  0   0  4
  [ 2] text              PROGBITS        00900000 000350 01232c 00  AX  0   0  8
  [ 3] .ARM.exidx        ARM_EXIDX       0091232c 01267c 000008 00  AL  2   0  4

Do you have any ideas on how to do that ?

#For IMX7D if code is not located in OCRAM_S
config CONFIG_ROMSTART_RELOCATION_ROM 
	default y if $(dt_chosen_reg_addr_hex,$(DT_CHOSEN_Z_FLASH) !=$(dt_nodelabel_reg_addr_hex,ocram_s_code)

What do think about it ?

[JiafeiPan]

Yes, there is a address mapping from M-Core address to A-Core address during loading from elf to memory, so address of 0x00000000 also works, but we still relocate rom_start to some specified address, 0x00000000 or 0x00180000, both are okay.

[jlh-makeen]

Exactly, that's the decision I took when making the PR, define the ocram_s of imx7d at address 0 instead of 180000 and tcm_l of imx6sx to 0 while providing a default relocation of the rom_start region towards address 0/1K region.

This ends up as a single config flag for both imx7d/imx6sx and should be a good template for other SOCs I guess.

[JiafeiPan]

This discussion has been addressed by PR: #71684, so let's close this discussion.