Proposal: Adding ESP32_SUBCLASS_PROJECT_TEMPLATE option

[linfan68]

I'm running into an situation where I need to include some native esp32 components into my project (the iot_usbh_modem component from esp-iot-solution)

The easy way to do this is to adding a file called "idf_component.yml" as follow:

dependencies:
  iot_usbh_modem:
    path: /workspaces/dk-hwjs/esp-iot-solution/components/usb/iot_usbh_modem
  # Notice I'm using a local copy of the component, normally it should be simplier like this:
  # component2:
  #   version: ">=2.0.0"

However, this requires updating the "Native" esp32 project, which is here:
$MODDABLE/build/devices/esp32/xsProj-esp32s3

Currently there is no good way to switch to a different "flavor" of the project template, this is hard-coded in:
$MODDABLE/tools/mcconfig/make.esp32.mk

PROJ_DIR_TEMPLATE = $(BUILD_DIR)/devices/esp32/xsProj-$(ESP32_SUBCLASS)

Here I'm proposing adding an optional build variable called ESP32_SUBCLASS_PROJECT_TEMPLATE, and change the code above into:

# if ESP32_SUBCLASS_PROJECT_TEMPLATE not defined, set it to ESP32_SUBCLASS
ESP32_SUBCLASS_PROJECT_TEMPLATE ?= $(ESP32_SUBCLASS)

PROJ_DIR_TEMPLATE = $(BUILD_DIR)/devices/esp32/xsProj-$(ESP32_SUBCLASS_PROJECT_TEMPLATE)

By doing this, simply adding this into the manifest file:

{
  "build": {
    "ESP32_SUBCLASS_PROJECT_TEMPLATE": "esp32s3_usb_modem"
  }
}

makes the template project switching possible.

This solves some problems but not all. In my case, I also need add idf_component.yml into the "copy file list" to make the make file copy this file, changes needed are (in make.esp32.mk file)

PROJ_DIR_TEMPLATE = $(BUILD_DIR)/devices/esp32/xsProj-$(ESP32_SUBCLASS_PROJECT_TEMPLATE)
PROJ_DIR_FILES = \
	$(PROJ_DIR)/main/main.c \
	$(PROJ_DIR)/main/component.mk \
	$(PROJ_DIR)/main/idf_component.yml \  # <<<< this line
	$(PROJ_DIR)/main/CMakeLists.txt \
	$(PROJ_DIR)/CMakeLists.txt \
	$(PROJ_DIR)/partitions.csv \
	$(PROJ_DIR)/Makefile

# 
#  and this
# 
$(PROJ_DIR)/main/idf_component.yml: $(PROJ_DIR)/main $(PROJ_DIR_TEMPLATE)/main/idf_component.yml
	cp -f $(PROJ_DIR_TEMPLATE)/main/idf_component.yml $@

[phoddie]

Your timing is great! I was planning to spend some time on this project in the coming days.

You seem to be taking a global approach -- e.g. there is one project template that is replaced to include all of the additional ESP32 native components. I'm not sure how that scales to a project composed of several modules, each of which needs to include a particular ESP32 native component. In that case, it could be convenient to have each module's manifest list the native modules needed rather than a single project-level file. The challenge then becomes how to collect that list of native modules and inject them into the build. What do you think?

[linfan68]

Yes you’re right, in an ideal world we should be able to inject native related changes by adding a module. However in my specific case (adding the usb component), I’ve tried to recover the full include list from the demo project, adding massive amount of build flags and options into he module but still failed. Also since there is a dependency chain of multiple modules needs to be imported. I think there are better ways (if I dig the native module build chain deeper), but adding one line in the idf_component.yml file just too simple so at that point I just gave up and hacked the template project….

[phoddie]

Hmmm..... OK. Nothing wants to be easy. ;) I'll keep exploring.

[linfan68]

I believe the idf_component.yml thing is pretty new in IDF. They are trying to build a package manager and package ecosystem.

A manifest generated idf_component.yml does help, but in my case I also needed to create a function in main.c to use features in the new component. I forget the exact reason… may due to hard to reproduce all include paths related to use headers of the new module.

So what I did was adding a “bridge” function in main function( like connectUsbModem()), then call that function from the native module, and JS code.

[linfan68]

Ha I found the original post I wrote:
#1245

In this case with the manifest adding include path ability, adding a manifest option that generates the yml file seems like a good approach to solve my problem.

However I still believe allowing modify the template project could be an ultimate fallback for adding new native features.

[phoddie]

Understood. It sounds like there may be a straightforward case that works in some (hopefully many) circumstances. But for some more complicated scenarios, more / complete control is required.