mqtt, value vs json

[xoseperez]

Originally reported by: f-fish (Bitbucket: f-fish, GitHub: Unknown)

---

Been playing with telegraf for data input into influxdb
Espurna seems to have always done a good job of single topic single value, and I suspect allot of the guys are using this to manage the devices. The domoticz integration is a bit more json like in the structure of the data, but effectively still 1 idx to a value.

In a time series db like influx I suspect a single entry with many topics for the same instance is more efficient and less io than a single line entry for each value pair.

I see as an example json reporting is the away some firmware is moving see https://groups.google.com/forum/#!topic/sonoffusers/84Io2UvWKWI but here the OP actually wants it to be value based.

Now the selfish part, to simplify integration with telegraf - processing a json string is way simpler and faster than all the individual mqtt events, it would also make for a single entry in your influxdb.

My ideal json would probably look something like this.

{"Time":"2017-02-19T21:59:46", "Host":"SONOFF_POW_E00FBB", "Yesterday":0.000, "Today":0.009, "Period":0, "Power":0, "Factor":0.00, "Voltage":0, "Current":0.000}

or

{"Time":"2017-03-19T20:00:36", "Host":"SONOFF_DS18B20_E00FBB", "RELAY":0, "LED":1, "IP":"192.168.1.22", "Version":"1.7.0", "Temperature":22.1, "VCC":3281}

The above would allow for very simple and efficient telegraf consumption.

#!arduino

  data_format = "json"

  ## List of tag names to extract from top-level of JSON server response
  tag_keys = [
    "Host",
    "IP",
    "Version"
  ]

See

https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md#json

When configuring

https://github.com/influxdata/telegraf/tree/master/plugins/inputs/mqtt_consumer

Not sure if this has ever been considered or requested. It could mean having to support yet another reporting format.

Later Ferdie

[xoseperez]

Original comment by f-fish (Bitbucket: f-fish, GitHub: Unknown):

---

Just revising my whist list for a good json ;-) based on enhancements in dev.

{"Time":"2017-03-19T20:00:36", "Host":"SONOFF_DS18B20_E00FBB", "RELAY":0, "LED":1, "IP":"192.168.1.22", "Version":"1.7.0", "Temperature":22.1, "VCC":3281, "Freeheap":30110, "Uptime":345443}

Later Ferdie

[xoseperez]

I've been thinking for a while about adding an option to report MQTT messages as JSON strings everywhere. For 1.X versions that would be a build flag (MQTT_USE_JSON). That flag would be read by any module reporting to MQTT to know the message format to use. Default value (MQTT_USE_JSON = 0) would be use individual messages for every magnitude to report. When set to 1 the messages would be grouped in a JSON object with some predefined headers (your suggestion is Time & Host).

The heartbeat message would behave like all the rest grouping the values (ip, version, vcc, heap, uptime and maybe relay status) is a single message.

[xoseperez]

Original comment by f-fish (Bitbucket: f-fish, GitHub: Unknown):

---

Sounds like a plan. Time and host makes some sense..

Time when the actual event happened - some tool can queue events before they hit the db's so then having an effective time stamps has some value. Host so that one can report on things easier and know at one glance who the data belongs to.

Happy with each event being jason - and yes the grouping the way to go on the bigger messages.

Later Ferdie

[xoseperez]

Original comment by Adil Bousfiha (Bitbucket: adil-bousfiha, GitHub: Unknown):

---

Hello,
I am interested in mqtt in json format,

Where is the configuration file with MQTT_USE_JSON?

Best regard

[xoseperez]

Original comment by Lauris Ieviņš (Bitbucket: laurisievins, GitHub: Unknown):

---

There isn't one yet.

[xoseperez]

Pushed with 1.8.3.

[xoseperez]

Removing milestone: 1.8.3 (automated comment)

[xoseperez]

Original comment by Fabien MARCHEWKA (Bitbucket: Fab_33, GitHub: Unknown):

---

Not all the data are present every time for example, relay/0 is present only when changing state.