Device Tree support overhaul

[carlescufi]

This is an umbrella issue for a series of changes to the way Zephyr uses Device Tree.
The following items need to be addressed in order to consider this issue closed:

ELCE meeting path forward issue: #10821

• Describe what is the scope of Device Tree. i.e. What should go into Device Tree and what into Kconfig
  • PR doc: dt: Clarify the relationship between DT and Kconfig #8507
  • Issue The guidelines for whether something should be in DTS or Kconfig are too vague #7230
• Run Device Tree before Kconfig
  • PR Shift DTS before Kconfig #12108
  • Issue DTS metadata should be made available to Kconfig #7302
• Device Tree improvements
  • Issue Generate DTS 'compatible' based compilation flags #9406: Generate DTS 'compatible' based compilation flags
• edts database generation (cut-out of scripts: add Zephyr inline code generation with Python #6762)
  • PR script/dts: Generate Extended Device Tree database #9876
  • https://github.com/zephyrproject-rtos/zephyr/tree/topic-EDTS branch
  • PR dts: Add new DTS/binding parser #17660
• Code generation for instancing of devices
  • PR Codegen: Generated driver instantiation prototype  #8561, scripts: inline code generation with cogeno (and edts) #10885, drivers: use codegen for device instantiation #10888 Jinja: [RFC] Template prototype using jinja2 templating engine #10957
  • Issue Requirements for driver devices generation using device tree #10904
  • https://github.com/zephyrproject-rtos/zephyr/tree/topic-EDTS branch
• (optional) Integrate DT support in menuconfig
  • PR
  • Issue

[SebastianBoe]

@erwango : I have replaced the codegen PR with your prototype PR under "Code generation for instancing of devices".

Is the prototype PR supposed to be a part of this effort? If so it should take into account the repercussions of running device tree before Kconfig. Or else we risk having to rewrite the whole thing again.

[erwango]

@SebastianBoe

Is the prototype PR supposed to be a part of this effort?

From what we discussed last week, one of the (/the) first step was to demonstrate the feasibility of using codegen to provide and acceptable solution for driver instantiation (from zephyr user point of view, the end user API). This is what #8561 aims at doing.
This task was supposed to go straight to the point, so I took whatever was available and tried to come up with a demonstrator. If there is a general agreement this is acceptable, then everything should be done the right way.

Doing so, there will be some additional efforts, but at least a plan is there and I hope showing something concrete will help people agreeing this is a desirable thing to have and efforts should be done to achieve the plan. (Rather than endless debates that happened so far in the various PR/issues)

[erwango]

Added #9406 as bullet of 'Run Device Tree before Kconfig '

#9604 proposes a way to provide a DTS generated set of flag that could be handled by Kconfig for driver build control.

[erwango]

Comment about: " Code generation for instancing of devices ":

Current PR for this point is #8561.
#8561 is based on #6762 that introduces Codegen support. It adds a friendly device declaration module on top of Codegen. It aims at hiding python code from the user in the driver code by providing a nearly python agnostic API.

[erwango]

Added #9876 as a cut out of #6762 (Codegen introduction)
#9876 focuses on edts database generation that should be used as input for codegen script.
It could also be used as input for "DTS compatibles flags generation" (#9604)

[b0661]

Sequence of merges for " Code generation for instancing of devices ":

• Code generation for instancing of devices

1. PR script/dts: Generate Extended Device Tree database #9876 - creation of extended DTS database using extract_dts_includes.py
2. PR scripts: add Zephyr inline code generation with Python #6762 - inline code generation - codegen
3. PRs for device driver instantiation by code generation
   • e.g. PR Generated driver instantiation prototype  #8561, drivers: use codegen for device instantiation #9163

[galak]

edts work handled by PR #17660

[mbolivar-nordic]

• Code generation for instancing of devices

Is anyone still advocating for this? I remain unconvinced that we need it and am opposed to the introduction of yet another domain specific language that this would imply for complexity reasons.

(optional) Integrate DT support in menuconfig

If nobody is still advocating for codegen and the above check box can be fixed, perhaps this can be moved to a separate enhancement and this issue can at last be closed. It isn't doing its job as an umbrella issue since we're not actively using it, and all the other items are checked.

[erwango]

@mbolivar-nordic Fine with me on both points!

[mbolivar-nordic]

Great, thanks @erwango. I've filed this issue for a 'menuconfig'-like (really 'guiconfig'-like) interface to devicetree here:

#29295

I think that covers the last unchecked box in this umbrella issue, so let's close it at last and discuss/prioritize additional work elsewhere.

[carlescufi]

• Code generation for instancing of devices

Is anyone still advocating for this? I remain unconvinced that we need it and am opposed to the introduction of yet another domain specific language that this would imply for complexity reasons.

Not really, but I do recall that there was some talk about whether pinctrl could effectively be implemented without code generation. Aside from that, the rest of the usecases (mainly instantiation of the device structures) I think are not justification enough to think about introducing this infrastructure.

[erwango]

Not really, but I do recall that there was some talk about whether pinctrl could effectively be implemented without code generation.

@carlescufi Since it is actually available at least on some targets (such as STM32 and Atmel) today, I think we can say this is doable.

[carlescufi]

Not really, but I do recall that there was some talk about whether pinctrl could effectively be implemented without code generation.

@carlescufi Since it is actually available at least on some targets (such as STM32 and Atmel) today, I think we can say this is doable.

Indeed, and the goal now is to extend that support to all SoCs in a common manner, thanks for commenting on this @erwango, this just confirms what I already thought.