flow-based programming

[eschava]

Hello

This is a proof-of-concept draft for a new functionality named "flow" that supports creating user scenarios executed right at the ESP8266 device running espurna firmware. Such scenarios are created using flow editor that is similar to node-red and others flow-based programming systems

Now the editor is implemented as a separate web application that should be run at the local computer (https://github.com/eschava/espurna-flow-editor) because it is large enough (13 MB). It uses AJAX requests to load/save current flow from/to espurna device.

Currently, only three types of nodes are implemented: MQTT subscribe, MQTT publish and Debug just to check that concept is working and evaluate interest to this feature

Flow is saved as JSON file at the SPIFFS storage so only devices having SPIFFS are supported (I use Shelly1 for my experiments)

Screenshot of flow editor is attached

To build firmware with all required functionality I used next flags:

-DWEB_REMOTE_DOMAIN="\"http://localhost:8080\""  
-DSPIFFS_SUPPORT=1  
-DFLOW_SUPPORT=1  

(WEB_REMOTE_DOMAIN is needed to allow AJAX requests to device from the remote site, by default it has http://tinkerman.cat value. I tried to host editor at the GitHub pages but it allows to use HTTPS only and it cannot access espurna device using HTTP due to security reasons)

I'm going to extend web flow editor with new features and implement new types of nodes (supporting buttons, relays, values comparison, etc) but created this pull request to get some feedback ASAP

[mcspr]

It is an actually cool approach at this 👍
(and something i never knew i wanted 🙄)

• Would it make sense to make broker generic event dispatcher here? i.e. like relays, send button state event and "handle" it via flow processor instead of modifying each module for flow support. sure, broker runs things in place right now, but it can be (and would probably will be, wip) modified to support delayed actions.
• I would be slightly worried about size limits for flow json. Any estimate on that?
• Minor espasyncwebserver quirk: I think beginResponse already can send out spiffs files more efficiently, no need to malloc them beforehand

And some notes about SPIFFS in general, as this is the first module using it:

• there are flash modules in the wild that do not work properly with it, see Rewrite PUYA patch to be more universal and mem friendly. esp8266/Arduino#5493. Only to worry about on 1M for now, but you never know.
• before 2.5.0 spiffs does not free it's buffers: SPIFFS.end() does not free memory esp8266/Arduino#4585

[eschava]

Would it make sense to make broker generic event dispatcher here?

I thought about some kind of broker but unfortunately, I don't know much about C++ and its memory management (I'm a Java developer) so local variables and in place calls is the best I can do :) For async actions component will be responsible for cloning variant object to process it later. But I'm open to any suggestions

I would be slightly worried about size limits for flow json. Any estimate on that?

Currently simple flow takes ~1 KB because uncompressed JSON is used. Version 6.0 of ArduinoJson library supports MessagePack format that is binary JSON and I think flow could be compressed up to 10 times after. And I hope in compressed format flow could be saved even to the EEPROM storage instead of SPIFFS.

Minor espasyncwebserver quirk

Thanks, applied the fix

[mcspr]

Baseline to know about c++ objects:
new allocates forever, needs delete
Object obj lives inside {}, destroyed afterwards
something(Object obj) copies the object, something(Object& obj) does not

Nice magic is "smart" pointers like shared_ptr / unique_ptr, that can hold new'ed objects. shared_ptr does ref-counting on copy (see copy constructors) and auto-deletes thing when everyone is done using it e.g. another object can hold it via copy and it lives with that object, even if {} block exit deleted the original one

And main thing to remember - there is only so much KiB of RAM for the whole app 🎐
(~20KiB in default configuration, maybe ~30KiB in barebones config. +4KiB using the newest Core)
Things like json parser will load up the whole file in ram. Unless some tricks are done, like positioning it on the json object boundary via manual seeking of the file object, because it stops parsing after the closing bracket.

[eschava]

Thanks for the information about C++ object allocation!
Currently, I'm thinking about porting this functionality to devices that have no SPIFFS storage, EEPROM only and need your help
I believe 1K should be enough for storing almost any flow using some compressed format
What API should I use? settings/EEPROM/EEPROM rotate/anything else?

[xoseperez]

Wow, this is actually very cool, like node-red embedded...
I'm mostly worried about the overall size of this and it's memory footprint too. I would not got the EEPROM way, there is no real advantage on using the EEPROM in the ESP8266, it's only a facade interface to access a flash sector but does not have wear leveling. EEPROM_Rotate only tries to mitigate this.
I will give it a try, hopefully tomorrow or tomorrow after and report back.

[gn0st1c]

back when i coded several versions of espurna scripting, overhead and memory was my concern too. so i settled for a simple linter.

basic.cpp.zip

[xoseperez]

Just for the record: I was also thinking on adding the RPNlib as an advanced schedule tool. It could easily be enhanced to perform actions on the relays based on time, sensor values, MQTT messages... But of course it's much more complex from the user point of view.

[eschava]

Thanks for the link. Flows definitely will need something like this for math calculations

Sorry for asking same question: what is the best way to store up to 1kb of binary data for devices having no spiffs storage?

[mcspr]

tldr; I'd use spiffs for now and not worry about it just yet. And build settings can be tweaked to make .bin smaller for some specific applications (like removing debug strings or web interface

It depends. Essentially, you only have EEPROM sector of 4KiB right at the end of usable flash that is expected to survive things like serial flash and OTA. Saving things in flash before the EEPROM is prone to overwriting by OTA, because flash writes are done in 4KiB blocks (For 1M, .bin is half the size most of the time)

First 306 bytes of EEPROM are reserved (see EEPROM_... defines in general.h), everything else is free-for-all and kind of prone to random overwriting.
Settings key-value storage is using EEPROM backwards, end-to-start (see eeprom.dump command. 2 bytes length, key, 2 bytes length, value), probably needs to be bound to something like 1.5KiB to not grow further than necessary.
Crash handler (see code/espurna/crash.ino) can have disableable stack trace recording.
This could free up at least 1KiB.

[eschava]

I've decided to use MQTT retained messages to store flow configuration if SPIFFS is not available

[eschava]

Guys, I've finished with all changes planned for the day one
Also, I registered http://espurna.online/ domain to have HTTP access to the https://github.com/eschava/espurna-flow-editor pages so currently extra flags are

-DWEB_REMOTE_DOMAIN="\"http://espurna.online\""    
-DFLOW_SUPPORT=1

(and plus -DSPIFFS_SUPPORT=1 for devices having SPIFFS memory)

Could you please review my changes and tell me if this feature could be merged to the main tree?

[mcspr]

Will check ASAP.

I would also add following env to the platformio.ini:

[env:nodemcu-lolin-flow]
platform = ${common.platform_latest}
framework = ${common.framework}
board = ${common.board_4m}
board_build.flash_mode = ${common.flash_mode}
lib_deps = ${common.lib_deps}
lib_ignore = ${common.lib_ignore}
build_flags = ${common.build_flags_4m1m} -DNODEMCU_LOLIN -DFLOW_SUPPORT=1 -DSPIFFS_SUPPORT=1 '-DWEB_REMOTE_DOMAIN="http://espurna.online"'
monitor_speed = ${common.monitor_speed}
extra_scripts = ${common.extra_scripts}

And there's crash when spiffs is empty. Looks like response handler needs to check if flow file exists first. Or some manual labor beforehand:

$ rm espurna/data/*
$ touch espurna/data/flow.json
$ pio run -e nodemcu-lolin-flow -t uploadfs

[mcspr]

Some things I should've mentioned earlier:

1. millis() value overflows in ~50 days, so direct comparison of time values will become invalid in scheduled callbacks for timer / delay
2. i think you can directly write into the spiffs file object instead of buffering it first, see webserver spiffs reader / writer files

And something I had missed from ArduinoJson 5 => 6 changes (which is likely to be used in the future instead of 5) - JsonVariant changed and can no longer work without backing memory pool. Implementation is small enough though: union {...} with _type field to determine what is inside. You can just search for "union" in the ArduinoJson code

[eschava]

fixed issue with millis() overflow

[mcspr]

Essentially, timers should calculate relative time.
now_millis - recorded_millis > interval
And am I understanding this will create ticker per input? Can it use only a single instance? I'll cancel old timer and start a new one

btw Ticker can only work with maximum value of 1h40m (tried some time ago to reimplement scheduler using it: #678 (comment))

[eschava]

And am I understanding this will create ticker per input?

Right, ticker per input, because there could be several inputs waiting in the queue and one ticker can handle only one timer.

Essentially, timers should calculate relative time.

I was going to slightly rework that code to do not use time comparison

[mcspr]

I see. Note that ticker function triggers in a system context, so does everything else in a process chain after that point. And with weird timing restriction, is it worth using it at all like that?
there are no problems right now, but using it inside loop() you can allow yourself to go back into the system via yield() / delay(), but not other way around

esp8266 arduino even has a nice abstraction for comparisons:
https://github.com/esp8266/Arduino/blob/master/cores/esp8266/PolledTimeout.h

[mcspr]

Amendment: new Ticker implements a special mechanism to execute function in a loop() context (aka CONT) by using schedule_function() (which I totally knew about...). see:
https://github.com/esp8266/Arduino/blob/64e30b270bcb8accde0c62d5998fd69907d764b6/libraries/Ticker/Ticker.h#L46
Schedule.h, .cpp

Note:
the hardcoded limit of 32 functions.
ticker functions not available in the 2.3.0
...or replicate them, place our handler at the end of the loop(), because a lot of Core libraries are using scheduled funcs

edit: i.e. it is probably better to reimplement such functionality - ticker callback queues the function, which is executed by handler running inside loop()