Feature: Conditions

[proddy]

By @MichaelDvP :

We have talked about conditions some time ago here, in scheduler discussion and when implementing custom variables. I now have a first implementation of conditions for the scheduler. My own use case is to use the relais of gateway V2 to control the solar assist valve, now done by ioBroker script and a sonoff module. To control i need a expression parser for logical and arithmetric functions.
relais on for mixer/hc2/pumpstatus == 1 && solar/cylbottomtemp > (temperaturesensor/boiler_backflow + 4)
relais off for mixer/hc2/pumpstatus == 0 || solar/cylbottomtemp <= temperaturesensor/boiler_backflow
First implementation looks like this:

and is working now (but still crashes if the condition could not be parsed correctly).

Another possible use case is for users without thermostat to implement a heatingcurve, than calculate the command value:

Also implemented is a possible trigger on change, e.g. the heatingcurve can be recalculated on every change of outdoortemp, not periodic.

I'll fix the crashes and make a feature-build for testing.

Originally posted by @MichaelDvP in #1704 (comment)

[proddy]

another use case could be
if rssi < -70 restart

[MichaelDvP]

if rssi < -70 restart

Difficult. We don't have get_value_info for SYSTEM-device. Using system/info and extract values is also nasty, because it is mostly human readable and have mixed case keys with blanks. We have to use:
condition: system/Network Info/RSSI < -70 to match, because arduinojson is case sensitve.
Actually i also check for blank to find the end of a command, Needs rework.
Problem is a arithmetic / after a command, custom/myVal / 2 > 5 will not work, it should be written as (custom/myVal) / 2 > 5

[MichaelDvP]

Ok, i've made a test-version with binaries:
https://github.com/MichaelDvP/EMS-ESP32/releases/tag/test

And these are the rules for conditions and arithmetics:

• a condition has to be a logic value 0 or 1, the condition is true only for 1, an arithmetric result 1 is also interpreted as true
  schedule command executed for 3 > 2, 3 - 2
  schedule command not executed for 3 < 2, 3 + 2
• blanks are not needed, but makes the formula more readable
• ems-esp values are accessed <device>/<entity> or <device>/<entity>/value, <entity> may contain prefixes like <hc2>.
• checking enum and bool values depends on api setting! Check the right value before setting a schedule
  e.g. check output http://ems-esp.local/api/thermostat to see building = "medium"to create the rule with thermostat/building == medium
  also for bool values check if you have to use 0/1, off/on, OFF/ON, or false/true
• strings containing special characters have to be written in quotations, e.g. boiler/pumpmode == "delta-P2", to avoid a calculation error on delta minus P2,
  other strings can be written with or without quotations
• commands followed by a divider / have to be set in parenthesis e.g. (boiler/seltemp)/2
• condition command is only executed on a change of the condition from false to true. If the condition stays true, the command is not repeated
• command value can also be a formula.
• actually the condition is checked every 10 sec. (Gives time to deactivate the schedule in case of esp crash, please report if there are crashes)
• allowed operations : arithmetric: + - * / % and prefix- logic: == != <= >= < > && || and prefix !
• usefull for the value of onChange and condition: a single instance of <cond1> ? <cond2> : <cond3>
• on change trigger is always one or more entitis <device>/<entity>, where device is not system. It can have multiple triggers. e.g. boiler/outdoortemp custom/setpoint and triggers on change of each value (entities never change the same time (it's a callback), logical operations like && makes no sense)

A few use cases are mentioned above. If users have other, please mention here, we can collect and add to the wiki.

[proddy]

Very excited about this feature! I would just do a PR on dev and let the community test.

[MichaelDvP]

PR done, dev.17

[proddy]

thanks, I'll review. With "actually the condition is checked every 10 sec. (Gives time to deactivate the schedule in case of esp crash, please report if there are crashes)" where is this defined? I couldn't spot it quickly.

[MichaelDvP]

see here: WebSchedulerService.cpp#L412-L417
I plan to use every second, after the first crash i changed quickly to 10s, but i think i've fixed all possible exceptions now.

[proddy]

@MichaelDvP shall I take your instructions above and add it to the EMS-ESP docs/wiki?

[MichaelDvP]

Yes, but i make some edits first and add:

• usefull for the value of onChange and condition: a single instance of <cond1> ? <cond2> : <cond3>
• trigger on change can have multiple triggers. e.g. boiler/outdorrtemp custom/setpoint and triggers on change of each value (entities never change the same time (it's a callback), logical operations like && makes no sense)

[proddy]

Yes, but i make some edits first and add:

ok let me know when you're done with the edits.

[MichaelDvP]

Already done., edited #1806 (comment)

I'm testing multiple instances of condition? expression1: expression2 eg. condition1? expression1: condition2 ? expression2 : expression3
But it's a first check before calculation the expressions and not working inside a formula
eg. 5 + (condition ? expression1 : expression2) is not allowed it should be condition ? 5 + expression1 : 5 + expression2

[proddy]

doc updated

[jannejman]

Good job. This new feature could help me unfortunately it doesn't work as I would have imagined.
I used the following parameters:

OnChange: boiler/watertapactive
Command: analogsensor/relay1
Value: boiler/watertapactive

Both watertapactive and relay1 use On/Off. The change should activate a relay to start water circulation when DHW heating is active (and stop when inactive). It works well for a timer set to 00:01, but it produces a lot of messages. The onChange feature should be better. Can I run a debug for the scheduler? I changed a log level but without success.

BTW: Is there any way to set multiple output values? For example, set relay1 on and set relay2 off.
Maybe it would be great to use <Input type="text"> for the Value, and then a list of actions to perform (one line one action). E.g. to set an additional flag (in RAM) that can be checked later.

[MichaelDvP]

Good catch. There are some special entities in ems-esp, that are not from ems and needs extra handling, i forgot to add them to scheduler.
I'll add boiler/heatingactive and boiler/tabwateractive to the trigger.
For shower it needs more work, we also don't have a shower-device and no /api/shower/... I'll skip the shower entities for now.

Multiple output commands and values are difficult in the actual structure, needs a lot of refactoring.
I'll make a change to allow same trigger in different schedules. So you can add an extra schedule for each command and use the same trigger.

[jannejman]

Thank you for the fix.

An extra schedule for each command is not equivalent to multiple actions because it doesn't guarantee action order.

Is it possible to add to the "system Info" time-related items? e.g. dow, day, month, year, hour, minute
It would be great to use these items in the onCondition formula. onCondition is actually an expert use of the scheduler. Comparison with time can improve the planner to a new level.
I can imagine something that runs a relay1 for 10 minutes once an hour (between 6 AM and 9 PM):
Scheduler1:
onCondition:
(system/time_hour >= 6) && (system/time_hour <= 21) && (custom/pumpactivelast/value + 3600 < system/uptime)
Commands:

analogsensor/relay1 = On
custom/pumpactivelast = system/uptime

Scheduler2:
onCondition: (analogsensor/relay1 == On) && (custom/pumpactivelast/value + 600 < system/uptime)
Commands:

analogsensor/relay1 = Off

[MichaelDvP]

See last point here #1806 (comment)
You can't use system entities for onChange, they are not updated frequently, only on request.
And there is never a garanteed order, the ems-devices needs time to execute commands, and they differ much.

[jannejman]

Michael, I was talking about onCondition, not onChange in the example above. If I understand correctly, the system items should be available in onCondition mode. Right? If onCondition is evaluated every 10 seconds, it might not be a performance issue.

[MichaelDvP]

Ah, sorry, yes, system entities are available in condition mode.

[MichaelDvP]

Completed.

[MichaelDvP]

@proddy Should we add for the schedule command a simple http request.
Maybe usefull to switch a wifi powerplug with tasmota or shelly api. This is safer than connection a 230VAC relais to a gpio.
But i fear users will soon ask for more complex apis and https :-(

This is what i've tested with tasmota plug:

bool WebSchedulerService::command(const char * name, const char * cmd, const char * data) {
    // check http commands. e.g.
    // tasmota(get): http://<tasmotsIP>/cm?cmnd=power%20ON
    // shelly(get): http://<shellyIP>/relais/0?turn=on
    if (strncmp(cmd, "http://", 7) == 0) {
        HTTPClient http;
        int        httpResult;

        http.begin(cmd);
        if (data && data[0] != '\0') { // post
            http.addHeader("Content-Type", data[0] == '{' ? "application/json" : "text/plain");
            httpResult = http.POST(String(data));
        } else { // get
            httpResult     = http.GET();
            String payload = http.getString();
            EMSESP::logger().info("http Result: %s", payload.c_str());
        }
        http.end();
        return httpResult > 0;
    }

[MichaelDvP]

I would make the HTTP non-blocking as I'm not sure if HTTPClient is asynchronous and it could cause a WDT if the response is not quick enough or too large.

I've tried with typos in url and non-existing IPs, no issue. Too large response could be an issue, i'm not sure if we need the response at all. For now i only use it to check if it works (log) (switching the tasmota plug of my desk-light is the second test).

Could you also add HTTP to the value? I.e get a value from the remote site?

Do you have an example for this? I think mostly the http get returns not a single value and needs further parsing.
Better create a custom ram value and use the remote site to set the value via API-command. In scheduler command value you can set the custom value.

[jannejman]

Example: I could read a value from another EMSESP system via API:
E.g. Value=http://192.168.1.127/api/boiler/dhw/seltemp/value
The API call returns just a value. You don't need to do any additional processing. Of course, it could be used only for a simple HTML response payload. For example, you can read a remote temperature from another system...

Do you plan to add HTTP requests to the onChange or onCondition types? Using math and conditions applied to the response payload would be nice. For example, you can set a dhw temperature based on the temperature of the secondary heater.

[MichaelDvP]

I don't think this is a real use-case. Yes, the ems-esp api can respond with a single value, but this is very uncommon.
As proddy mentioned, waiting for reponse could be timing critical. Also parsing the url can conflict with formula parsing.
It's not worth the effort.

I only want to add it as single command, just http, no response checking, no https, no authentification, etc.

I've found only one usecase: building a useless box with my desk light:

[proddy]

Another idea, more complex, is to create a new command (called REST or HTTP for example) that has as its value/data JSON these 4 keys: a URL, a Type which is either post or get, a Header string and a body.

That way we can call Home Assistant services and scripts. I tested from a command line calling HA with:

curl -X POST \
    ${ha_url}/api/services/script/test_notify \
    -H "Authorization: Bearer ${ha_token}" \
    -H "Content-Type: application/json"

[jannejman]

Michael, another hypothetical example (it's not my case): you have two boilers, usually only one is running, but if necessary the other one comes on. This is either due to power needs or simply for maintenance. If each boiler had its own EMSESP controller, then the value set by remote status would be a good feature.

The backup boiler would just check the status of the primary, and if it was in the off state, or the return temperature was low, the second boiler would come on... This could then be easily done via that HTTP request to the API of the primary boiler. This way there would not even be a need for HA. Two or more boilers are often used for apartment buildings, but as I wrote, this is not my case, it could be quite a useful feature.

It seems to me that the scheduler is getting quite complex, and there could be difficulties in distinguishing what is an ESP command, what is a URL, what is a string, and what is a number. Just wondering if it would like some refactoring, and to distinguish between a command and a string. I know that commands can be parenthesized, but in some cases, the parenthesized command didn't work for me. This is not a criticism of the code, currently, the scheduler does everything I need it to do, but it's just a thought for future extensions...

I like Prody's example, but it adds complexity to specifying parameters again. It's hard to put everything that into just the Condition, Command, and Value triplet. If you plan on extending the scheduler, it might be nice to consider some kind of expert scheduler setup. Something like a script?

[MichaelDvP]

@proddy Yes, this could be a way to add the authorization header.
I think this should work:

• as in my approch: if value field is empty, use GET, otherwise use POST, value filed always contains the http-body.
• use json to add headers. Command field is then {"url":"http://rest.of.url", "header":{"authorization":"Bearer xxx", "Content-Type":"text/plain"}}

@jannejman Not a convincing example, in this case you should mount a ems-cascade module (MC400).

[proddy]

@MichaelDvP I like it. Let's do it. It'll make EMS-ESP very powerful

[MichaelDvP]

I've added it to the PR, bin is https://github.com/MichaelDvP/EMS-ESP32/releases

[proddy]

the http works well. I was able to call Home Assistant scripts via the scheduler, which is fantastic. I'll merge the PR, and we can continue to enhance it. Two changes needed

• to be able to POST without sending a data payload. This is needed for HA, and EMS-ESP uses the same logic, using POST to force bearer authentication for admin-only calls. Perhaps check the Value for a special code like '-' ?
• if value starts with a '{' we can automatically set the header to "Content-Type: application/json" which saves space in the header.
• changes to the Web to allow URL strings > 64. A call to HA with the bearer token is 290 chars. https://developers.home-assistant.io/docs/api/rest/#post-apiservicesdomainservice

I'll make some changes.

[MichaelDvP]

Agree, but special code is not very intuitive, I suggest:

                String methode = doc["methode"] | "";
                if (value.length() || methode == "post") {
                    httpResult = http.POST(value);

I think setting a header should have priority, so check:

                bool content_set = false;
                for (JsonPair p : doc["header"].as<JsonObject>()) {
                    http.addHeader(p.key().c_str(), p.value().as<String>().c_str());
                    content_set |= (p.key() == "content-type");
                }
                String value = doc["value"] | "";
                if (value[0] == '{' && !content_set) {
                    http.addHeader("Content-Type", "application/json");
                }

[proddy]

yeah, I had already coded it like below, before seeing your response. we're close:

 if (c) { // parse json
        JsonDocument doc;
        int          httpResult = 0;
        if (DeserializationError::Ok == deserializeJson(doc, c)) {
            HTTPClient http;
            String     url = doc["url"];
            if (http.begin(url)) {
                // It's an HTTP call

                // add any given headers
                for (JsonPair p : doc["header"].as<JsonObject>()) {
                    http.addHeader(p.key().c_str(), p.value().as<String>().c_str());
                }
                String value  = doc["value"] | data;   // extract value if its in the command, or take the data
                String method = doc["method"] | "GET"; // default GET

                // if there is data, force a POST
                if (value.length()) {
                    if (value[0] == '{') {
                        http.addHeader("Content-Type", "application/json"); // auto-set to JSON
                    }
                    httpResult = http.POST(value);
                } else {
                    // no value, but check if it still a POST request
                    if (method == "POST") {
                        httpResult = http.POST(value);
                    } else {
                        httpResult = http.GET(); // normal GET
                    }
                }

                http.end();

                // check HTTP return code
                if (httpResult != 200) {
                    char error[100];
                    snprintf(error, sizeof(error), "Schedule %s: URL command failed with http code %d", name, httpResult);
                    emsesp::EMSESP::logger().warning(error);
                    return false;
                }
                return true;
            }
        }
    }

[MichaelDvP]

I have made the scheduler loop async now. takes ~6k ram with 4k stack set. Testing now.

[proddy]

nice! I've just merged into some small changes

[proddy]

I think there may be a memory leak somewhere with the HTTP calls - haven't looked at it yet. on my E32v2 free mem dropped from 175 KB to 43KB. Same with max alloc.

With no scheduler items, free mem is now at 137 KB on dev-26, so 38KB is being used up somewhere.

[MichaelDvP]

What schedules were you running? How long?
I had a GET in condition running all night without memory drop. But no comand executed (condition false).
I start now with get in condition returning true and do a post in command.

[proddy]

I don't have anything in the scheduler - no entries. After reboot, within 2 minutes you can see the memory dropping. I'll also take a look later.

[MichaelDvP]

Is this with onChange queing strings or the actual dev?

[proddy]

I think it must have been an issue with the upload script I was using. I'm taking now the .bin from GitHub CI and its working fine with 166 KB (107 KB max alloc). Sorry for the false alarm.

I did notice with an E32V2 clicking on the web download downloads the wrong file (not the _16M) so I'll fix that

[MichaelDvP]

I did notice with an E32V2 clicking on the web download downloads the wrong file (not the _16M) so I'll fix that

Oh, i think that's what happend to @tp1de.

I've tried if it save memory if we load HTTPClient dynamic, e.g.

HTTPClient* http == nullptr;
...
if (http == nullptr) {
    http = new HTTPClient;
}
http->begin(url);
...
http->end();
delete http;
http = nullptr;

but can not see any difference in heap/maxAlloc. Tested on a standalone S3 without bus-connection, only activated or deactivate schedules with http get/post.

[tp1de]

I did notice with an E32V2 clicking on the web download downloads the wrong file (not the _16M) so I'll fix that

Oh, i think that's what happend to @tp1de.

Yes, I can confirm (I haven't recognized). I need to flash the gateway to get it working again.

[proddy]

I think we can close this, it works fine unless you can think of more things we can add or improve?

[MichaelDvP]

Yes, i close it for now.
The more thing i thinking about is acually a command:
{"mailto":"[email protected]", "subject":"from emesep", "message":"something happend"} (o message in value field)

• We need ssl/starttls (should work on 16M vsersions)
• we need smtp server config with user/pass. Can be added in the json or add a permanent config to settings/application.
• with permanent config we can make some checkboxes: [ ] message on reboot, [ ] message on mqtt disconnect, etc. But this could also be done in scheduler (reboot: timer 00, mqtt: condition system/mqtt/status != "connected"

There is still a lot to think about.
We can reopen after stop thinking and starting coding.

[proddy]

Yeah, I always like the idea of email (copying the feature from https://husdata.se/docs/h60-manual/notifications/). Pretty sure we discussed it before, but can't remember where.

We have the system/message function now, so we could extend with a type (mqtt, email, both) and take an optional address and subject in an optional JSON body. Maybe call it Notifications ;-)

The SMTP can be added to settings, like my Unifi network does:

This would be a new Feature GitHub issue, we can still keep this closed and play around in a separate fork.

If

[MichaelDvP]

Oh, i have not husdata in mind, The notification is a feature of a lot of 24/7 deviceslike routers, NAS, etc. Also tasmota have it. https://github.com/arendst/Tasmota/blob/development/tasmota/tasmota_xdrv_driver/xdrv_01_1_webserver_mail.ino

But email is for people today not fast enough, They want push notification like whatsapp, or sililar and there are a lot of web services (payed) that can be used via http(s). (Still thinking)

[proddy]

I use HA's notify to send Push messages that get triggered on EMS-ESP events. I've also used Pushover before. We can use it now using their REST interface and the nice work you did on the Command handling HTTP calls. The API is similar to HA's, just need to provide a Token. https://pushover.net/api

[MichaelDvP]

Tested the pushover now. Nice.
The message has to be inside a json, some small changes needed to make it work with custom message:

This pushesthe message each time the custom/message is updated.

Put this in wiki as tips&tricks or in scheduler section?