API v3 changes

[proddy]

From the discussion at https://github.com/emsesp/EMS-ESP32/discussions/61 and the previous work done in #50 here's the new specification. Coding will be done in the ft_apiv3 feature branch. This issue will be updated to reflect any changes.

---

Commands

• Commands are actions. They execute a particular function and return a JSON response. For examples of the OpenAPI format see how GitHub does this.
• The System context has the commands send, publish, fetch, and reset. And these are top-level, so there is no need for a System context anymore.
• EMS device value names (e.g. seltemp, selflowtemp) are not commands as they were in v2.
• commands have a description, although currently not used.

via console:

• Commands can be called from the Console using the call command. $ call <context> <command> <data> where <context> is either system or dallasensor or an EMS device like boiler. e.g. $ call system publish`` or $ call system send "12 13 14".
• Commands are either public or private. Private requires an authorization token in the API or be su in the Console.
• It's Important to note that some specific Console commands and not the same as commands. For example show set log help watch are specific only to the telnet console.

via REST API:

• commands are exposed as API endpoints, e.g. POST /api/send and take an optional JSON body. They only use HTTP POST and authentication is required, either as a Bearer HTTP header or as a access_token query parameter in the URL.
• Note the 'system' context is no longer needed. These are now all commands at root level.
• The JSON body is specific to the command, for example api/pin will take a JSON body {gpio: <number>, value: <boolean>}.
• Important to note some Console commands are specific to the Console, such as show and note exposed as regular commands. The show command will display all EMS and Dallas Sensor data in short-name format (see below).
• On a successful command execution the JSON message is HTTP/1.1 401 Unauthorized {"message": "OK"}
• With invalid JSON parameters it will return HTTP/1.1 400 Bad Request {"message":"Problems parsing JSON"}
• With incorrect or missing authorization it will return HTTP/1.1 401 Unauthorized {"message": "Bad credentials"}

Device & Device Values

• Each EMS device has a set of known device values. For each device value we donate a short-name (e.g. seltemp) and a full-name ("Selected Temperature"). The full-name is what is shown in the Web UI.
• Device value also have a UOM (unit of measure) and either a group or range (min, max values)
• Device values can be hidden. If hidden they are used internally or for MQTT, but not shown in the Web, Console or API calls. For example temp or hatemp or hamode which is required for Home Assistant.

via console:

• Device values can be queried from the Console using $ call {device} {name} <value> e.g. call thermostat seltemp 20. The value is transposed in the code to a JSON object {"value":<value>} to keep to the same API standard.

via REST API:

• Each device value is exposed as an API endpoint in the form /api/devices/{device}/:name. name is the short-name.
• For some devices like thermostats and mixers an additional hc or wwc path can be added in the URL, e.g. /api/devices/thermostat/hc2
• An HTTP GET to the endpoint returns the value and its characteristics. This is public and does not require authorization. e.g. GET /api/devices/thermostat/hc1/seltemp. If the name is omitted then all the "name:value" pairs will be listed (e.g. GET /api/devices/thermostat).
• GET /api/devices lists all devices
• An HTTP POST to the endpoint is for updating a specific value and takes a JSON body with the required data. All calls require authorization either in the HTTP Header with "Authorization: Bearer {ACCESS_TOKEN}" or using the access_token query parameter in the URL. e.g. POST /api/devices/thermostat/hc1/seltemp with a body {"value":20} will set the room temperature to 20 degrees.
• The JSON body is specific to the device and the value, for example, a thermostat could take an optional heating circuit "hc":<number>" and a mixing unit a warm water circuit number "wwc":<number>.
• Although not sure if we still need wwc - to be checked
• The old v2 API with having cmd, data and id as query parameters ([http://ems-esp/api?device](http://ems-esp/api?device=)=<device>&cmd=<command>&data=<data>&id=<id>) is no longer supported. Note, this will be a breaking change.
• Invalid JSON parameters will return HTTP/1.1 400 Bad Request {"message":"Problems parsing JSON"}
• Successful operations return HTTP/1.1 200 Bad Request {"message":"OK"}

Dallas Sensors

• Major change is the rename from dallassensor to sensor.

via console:

• via the show command the sensor data will be listed. For example show or show sensors.

via REST API:

• The endpoint is /api/sensors and returns all the sensor values

Summary of API endpoints

GET to fetch data, POST with authentication token/header and a JSON body to set data.

• /api/devices (GET only)
• /api/devices/thermostat[/hc<n>]/:name
• /api/devices/boiler/:name
• /api/devices/mixer/:name
• /api/devices/solar/:name
• /api/sensor:sensorid (GET only)
• /api/{command} (POST only)

over MQTT

• MQTT subscriptions work in the same way and match the REST API structure using the same path without the /api/. For example clients can subscribe to ems-esp/devices/thermostat and send a payload with a JSON {name:"seltemp", "hc":2,"value":22} and also ems-esp/devices/boiler/selflowtemp with a payload {"value":60}.
• Devices with circuits such as thermostats and mixers can have a circuit in the path just like the API, e.g. MQTT subscribes to the topic ems-esp/devices/thermostat/hc2.
• The MqttSubFlag is probably no longer needed

To change

• remove pre-processing the commands and names, like checking for "hc<>n."
• remove the commands info_short, info and commands
• add a reset command
• remove device names from registering as commands
• removing handling of the query parameters cmd, id, data in the URL
• authentication can still be bypassed by the Web setting
• new endpoint api/devices that lists all the devices and their characteristics (not values)
• endpoint api/devices/{device} to return all values and properties of a device
• new endpoint api/sensors to list all the sensors are their values
• look into reducing the length of the token from 128 bytes. SHA-256 is 32 chars.
• add a timeout to the GET HTTP header
• clean-up JSON return errors (message, code) to be consistent
• remove the version from hashing algorithm so access codes are not altered between releases
• all the C++ functions take only a JSON body as a parameter and not the traditional value/data/id group which also simplifies the code
• remove detecting wwc in URL - check if needed for Mixer?
• remove system context. In API for example /api/system/send becomes /api/send
• change MQTT to no longer subscribe to ems-esp/system but just ems-esp
• implement show sensor and the /api/sensor endpoint for the Dallas Sensors
• remove the MqttSubFlag from commands and use intuitive wording in Web UI
• tidy up the documentation at https://emsesp.github.io/docs/#/APIv3

[MichaelDvP]

There are some things in the spec i don't understand. I think i'll wait for the code and look into it and test.
Or you can clearify in a few words. Don't wast time to rewrite the spec.

EMS device value names (e.g. seltemp, curflowtemp) are not commands as they were in v2.

curflowtempwas always a value, never a command, not in v2, not in v3. seltemp was no command in v2, it was added in v3 to have in web a value/command combination to edit and it is the same command as temp in v2. So i don't know what the above statement tells me and what is the change.

Device values can be queried from the Console using $ call {device} {name} e.g. call thermostat seltemp 20.

I thinks this should be for commands, not for value query? Or do i really send a value to query a value?

Each device value is exposed as an API endpoint in the form /api/devices/{device}/:name. name is the short-name.

Do we really need the extra /devices/ in each path?

added in the URL, e.g. /api/thermostat/hc2

Here without the /devices/, looks better to me, but should be always one syntax.

[mqtt]

and also ems-esp/devices/boiler/curflowtemp with a payload {"value":60}.

I thought the direct commands in mqtt stay as single value, but now you are back to json-payload. I think direct subscriptions are useless if they need a json. If i want to use a json i can also write to /boiler. (btw. currflowtemp isn't a command). Again here you use the extra /devices/, for what?

BTW: Inspired by the single subscriptions i added single value publish on change. Works and (surprise) don't flood the mqtt-queue.
I can add as a ft-branch, but i think i'll wait for the command changes. I think it will conflict.
Here i a screenshot of mqtt receive with timestamps:

[proddy]

Thanks for reviewing. I haven't started any programming yet - still thinking about it and I want to get to 100% correct

curflowtempwas always a value, never a command, not in v2, not in v3. seltemp was no command in v2, it was added in v3 to have in web a value/command combination to edit and it is the same command as temp in v2. So i don't know what the above statement tells me and what is the change.

register_device_value() adds each device value to the Command list via Command::add() . I want to keep the cmdfunctions_ vector only for commands. A device value can have an associated callback function but will be handled differently so there is a clear separation between real "commands" and values.

Device values can be queried from the Console using $ call {device} {name} e.g. call thermostat seltemp 20.
I thinks this should be for commands, not for value query? Or do i really send a value to query a value?

Good catch. So $ call only to call commands which take an optional JSON argument ("{ .... }"). And to change a value of a device value then perhaps just $ {device} {name} {value} e.g. $ thermostat seltemp 20 ? This matches the logic in the Web too.

Do we really need the extra /devices/ in each path?

Yes I prefer so. It keeps it clean as we may add other endpoints, like api/users, api/sensors etc. It also keeps in line with the OpenAPI specification.

I thought the direct commands in mqtt stay as single value, but now you are back to json-payload. I think direct subscriptions are useless if they need a json. If i want to use a json i can also write to /boiler.

What do you suggest for the subscriptions? Apart from the special case of MQTT should we allow device values to be set via MQTT publishes? If so how since there almost 200 device values?

[MichaelDvP]

register_device_value() adds each device value to the Command list via Command::add() . I want to keep the cmdfunctions_ vector only for commands. A device value can have an associated callback function but will be handled differently so there is a clear separation between real "commands" and values.

register_device_value() is overloaded, the plain function only adds the value and the bool has_cmd flag. Only if it is called with a command functions it splits in command::add for the command and register_device_value (with bool flag) for the value. The command-vector only holds commands, the devicevalue-vector only devicevalues. I think there is nothing mixed, only the function name of the overloaded function.

Yes I prefer so.

So api/system/send changes to api/send but terminal call system send stays with context system?
Also api/thermostat/hc2/mode changes to api/devices/thermostat/hc2/mode but terminal call thermostat mode
Also mqtt is different, now we have mqtt topic: system, payload {"cmd":"send", "data":"xx"}, should this go to topic send with what payload?

I prefer to have similar syntax in telnet, mqtt and api.

What do you suggest for the subscriptions? Apart from the special case of MQTT should we allow device values to be set via MQTT publishes? If so how since there almost 200 device values?

What we have now is the optional subscription of all commands, with all hcs it's a lot, as discussed in #31:

• standard:
  mqtt topic thermostat, payload {"cmd":"mode", "data":"auto", "hc":2}
  BTW: we have "data" in mqtt, but "value" in api, this should be the same keyword.
• direct:
  with direct full subscriptions topic thermostat/hc2/mode, payload auto
  with direct single subscriptions topic thermostat/mode, payload auto
  The direct subscriptions don't need a json, they work like the thermostat_hc subscriptions for HA. I think this is a great advantage!

Do you also want to change the mqtt to subpath /devices/, like /devices/thermostat/hc2/mode`?

What i additional suggest (as feature-branch to test) is, if subscribing to direct topics, also publish the direct values, that's a lot more. But timing comes from ems-bus, on receiving (has_update()) a telegram the values that has changed in this telegram are publishes immediatly, and this are only a few values at a time.
Example:
Having thermostat in daymode 22°C and nighttemp set to 17. Writing to subscription thermostat/hc2/mode payload night will give a publish from ems-esp of thermostat/hc2/seltemp 17 and thermostat/hc2/mode night. But no other values are published.
I add a small 2 min log, you can see that it's not to much publish in normal operation.
20210518_143611_EMSlog.txt

[proddy]

register_device_value() is overloaded, the plain function only adds the value and the bool has_cmd flag. Only if it is called with a command functions it splits in command::add for the command and register_device_value (with bool flag) for the value. The command-vector only holds commands, the devicevalue-vector only devicevalues. I think there is nothing mixed, only the function name of the overloaded function.

I'm pretty sure selflowtemp is a Command now (can't check code). And if so, it's wrong and that's my whole point.

So api/system/send changes to api/send but terminal call system send stays with context system?

There is no system anymore, anywhere.

Also api/thermostat/hc2/mode changes to api/devices/thermostat/hc2/mode but terminal call thermostat mode

Again, call is only for commands. mode is not a command. It's a device value!

Also mqtt is different, now we have mqtt topic: system, payload {"cmd":"send", "data":"xx"}, should this go to topic send with what payload?

This all needs to change yes. I described in the specification one solution using a command endpoint. Again, forget system - it's dead!

I prefer to have similar syntax in telnet, mqtt and API.

That is my number 1 goal. It's hard though. I'd rather have the API lead, then MQTT follow and the Console is really for power users and for debugging, so doesn't need to be fancy. How often are people going to change the thermostat temperature from a Telnet console?

The direct subscriptions don't need a json, they work like the thermostat_hc subscriptions for HA. I think this is a great advantage!

yes, but still not 100% sure what you mean and how to get around having many subscriptions. I'll wait for the PR/feature branch and try it out. Also worth asking do we really need that level of control via MQTT now we have a more robust API? MQTT is great for capturing data and a REST API is better at calling commands. The same as with Home Assistant.

[MichaelDvP]

I'm pretty sure selflowtemp is a Command now (can't check code).

Sure it is, there was discussion #43. Exactly it is only value without command for cascaded heatingsources:

register_device_value(TAG_HS1 + hs, &selFlowTemp_, DeviceValueType::UINT, nullptr, FL_(selFlowTemp), DeviceValueUOM::DEGREES);

and value and command for the main heating source.

register_device_value(TAG_BOILER_DATA, &selFlowTemp_, DeviceValueType::UINT, nullptr, FL_(selFlowTemp), DeviceValueUOM::DEGREES, MAKE_CF_CB(set_flow_temp));

And if so, it's wrong and that's my whole point.

At this point i can't follow. Why is it wrong? How to change the flowtemp without a command?

[mqtt]

yes, but still not 100% sure what you mean and how to get around having many subscriptions.

The command subscriptions are in the code since #31, in dev and main branche, setting Subscribe format, individual topics in mqtt-setting page. These subscriptions work with single value, not with json.

My enhancement is publishing all data as individual values, each with it's own topic and a single value. These are no additional subscriptions, it's only publishing, but a lot of values.

Also worth asking do we really need that level of control via MQTT now we have a more robust API? MQTT is great for capturing data and a REST API is better at calling commands.

I prefer the mqtt, api has to be polled, mqtt can publish on change.

[proddy]

ok. I'll leave it. Too much of a headache and I have other things I need to work on

[proddy]

most of this was added to #173. A few things didnt make it or were not necessary. Such as the HTTP timeout is something that the client needs to add to the request (not EMS-ESP) and the size of the JWT passcode is according to the spec on https://jwt.io/ and needs to be large since EMS-ESP is not a valid Auth server.