Add json_key register setting for #9 (#22)

* Reformat readme
* Support for json_key register field allowing for multiple values to be published as JSON to a single MQTT topic.

README.md

@@ -48,19 +48,16 @@ address_offset: 0
 variant: sungrow
 scan_batching: 100
 ```
-
-`ip` (Required) The IP address of the modbus device to be polled. Presently only modbus TCP/IP is supported.
-
-`port` (Optional: default 502) The port on the modbus device to connect to.
-
-`update_rate` (Optional: default 5) The number of seconds between polls of the modbus device.
-
-`address_offset` (Optional: default 0) This offset is applied to every register address to accommodate different Modbus addressing systems. In many Modbus devices the first register is enumerated as 1, other times 0. See section 4.4 of the Modbus spec.
-
-`variant` (Optional) Allows variants of the ModbusTcpClient library to be used. Setting this to 'sungrow' enables support of SungrowModbusTcpClient. This library transparently decrypts the modbus comms with sungrow SH inverters running newer firmware versions.
-
-`scan_batching` (Optional: default 100) Must be between 1 and 100 inclusive. Modbus read operations are more efficient in bigger batches of contiguous registers, but different devices have different limits on the size of the batched reads. This setting can also be helpful when building a modbus register map for an uncharted device. In some modbus devices a single invalid register in a read range will fail the entire read operation. By setting `scan_batching` to `1` each register will be scanned individually. This will be very inefficient and should not be used in production as it will saturate the link with many read operations.
-
+| Field name | Required | Default | Description |
+| ---------- | -------- | ------- | ----------- |
+| ip | Required | N/A | The IP address of the modbus device to be polled. Presently only modbus TCP/IP is supported. |
+| port | Optional | 502 | The port on the modbus device to connect to. |
+| update_rate | Optional | 5 | The number of seconds between polls of the modbus device. |
+| address_offset | Optional | 0 | This offset is applied to every register address to accommodate different Modbus addressing systems. In many Modbus devices the first register is enumerated as 1, other times 0. See section 4.4 of the Modbus spec. |
+| variant | Optional | N/A | Allows variants of the ModbusTcpClient library to be used. Setting this to 'sungrow' enables support of SungrowModbusTcpClient. This library transparently decrypts the modbus comms with sungrow SH inverters running newer firmware versions. |
+| scan_batching | Optional | 100 | Must be between 1 and 100 inclusive. Modbus read operations are more efficient in bigger batches of contiguous registers, but different devices have different limits on the size of the batched reads. This setting can also be helpful when building a modbus register map for an uncharted device. In some modbus devices a single invalid register in a read range will fail the entire read operation. By setting `scan_batching` to `1` each register will be scanned individually. This will be very inefficient and should not be used in production as it will saturate the link with many read operations. |
+
+### Register settings
 ```yaml
 registers:
   - pub_topic: "forced_charge/mode"
@@ -83,24 +80,25 @@ registers:
   - pub_topic: "first_bit_of_second_byte"
     address: 13001
     mask: 0x0010
+  - pub_topic: "load_control/optimized/end_time"
+    address: 13013
+    json_key: hours
+  - pub_topic: "load_control/optimized/end_time"
+    address: 13014
+    json_key: minutes
 ```
 
 This section of the YAML lists all the modbus registers that you consider interesting.
 
-`address` (Required) The decimal address of the register to read from the device, starting at 0. Many modbus devices enumerate registers beginning at 1, so beware.
-
-`pub_topic` (Optional) This is the topic to which the value of this register will be published.
-
-`set_topic` (Optional) Values published to this topic will be written to the Modbus device.
-
-`retain` (Optional: default false) Controls whether the value of this register will be published with the retain bit set.
-
-`pub_only_on_change` (Optional: default true) Controls whether this register will only be published if its value changed from the previous poll.
-
-`table` (Optional: default 'holding') The Modbus table to read from the device. Must be 'holding' or 'input'.
-
-`value_map` (Optional) A series of human-readable and raw values for the setting. This will be used to translate between human-readable values via MQTT to raw values via Modbus. If a value_map is set for a register the interface will reject raw values sent via MQTT. If value_map is not set the interface will try to set the Modbus register to that value. Note that the scale is applied after the value is read from Modbus and before it is written to Modbus.
-
-`scale` (Optional: default 1) After reading a value from the Modbus register it will be multiplied by this scalar before being published to MQTT. Values published on this register's `set_topic` will be divided by this scalar before being written to Modbus.
-
-`mask` (Optional: default 0xFFFF) This is a 16-bit number that can be used to select a part of a Modbus register to be referenced by this register. For example a mask of `0xFF00` will map to the most significant byte of the 16-bit Modbus register at `address`. A mask of `0x0001` will reference only the least significant bit of this register.
+| Field name | Required | Default | Description |
+| ---------- | -------- | ------- | ----------- |
+| address | Required | N/A | The decimal address of the register to read from the device, starting at 0. Many modbus devices enumerate registers beginning at 1, so beware. |
+| pub_topic | Optional | N/A | This is the topic to which the value of this register will be published. |
+| set_topic | Optional | N/A | Values published to this topic will be written to the Modbus device. Cannot yet be combined with json_key. See https://github.com/tjhowse/modbus4mqtt/issues/23 for details. |
+| retain | Optional | false | Controls whether the value of this register will be published with the retain bit set. |
+| pub_only_on_change | Optional | true | Controls whether this register will only be published if its value changed from the previous poll. |
+| table | Optional | holding | The Modbus table to read from the device. Must be 'holding' or 'input'. |
+| value_map | Optional | N/A | A series of human-readable and raw values for the setting. This will be used to translate between human-readable values via MQTT to raw values via Modbus. If a value_map is set for a register the interface will reject raw values sent via MQTT. If value_map is not set the interface will try to set the Modbus register to that value. Note that the scale is applied after the value is read from Modbus and before it is written to Modbus. |
+| scale | Optional | 1 | After reading a value from the Modbus register it will be multiplied by this scalar before being published to MQTT. Values published on this register's `set_topic` will be divided by this scalar before being written to Modbus. |
+| mask | Optional | 0xFFFF | This is a 16-bit number that can be used to select a part of a Modbus register to be referenced by this register. For example a mask of `0xFF00` will map to the most significant byte of the 16-bit Modbus register at `address`. A mask of `0x0001` will reference only the least significant bit of this register. |
+| json_key | Optional | N/A | The value of this register will be published to its pub_topic in JSON format. E.G. `{ key: value }` Registers with a json_key specified can share a pub_topic. All registers with shared pub_topics must have a json_key specified. In this way, multiple registers can be published to the same topic in a single JSON message. If any of the registers that share a pub_topic have the retain field set that will affect the published JSON message. Conflicting retain settings are invalid. The keys will be alphabetically sorted. |

modbus4mqtt/modbus4mqtt.py

@@ -1,6 +1,7 @@
 #!/usr/bin/python3
 
 from time import sleep
+import json
 import logging
 from ruamel.yaml import YAML
 import click
@@ -77,6 +78,10 @@ def poll(self):
             self.connect_modbus()
             return
 
+        # This is used to store values that are published as JSON messages rather than individual values
+        json_messages = {}
+        json_messages_retain = {}
+
         for register in self._get_registers_with('pub_topic'):
             try:
                 value = self._mb.get_value(register.get('table', 'holding'), register['address'])
@@ -98,8 +103,22 @@ def poll(self):
                 if value in register['value_map'].values():
                     # This is a bit weird...
                     value = [human for human, raw in register['value_map'].items() if raw == value][0]
-            retain = register.get('retain', False)
-            self._mqtt_client.publish(self.prefix+register['pub_topic'], value, retain=retain)
+            if register.get('json_key', False):
+                # This value won't get published to MQTT immediately. It gets stored and sent at the end of the poll.
+                if register['pub_topic'] not in json_messages:
+                    json_messages[register['pub_topic']] = {}
+                    json_messages_retain[register['pub_topic']] = False
+                json_messages[register['pub_topic']][register['json_key']] = value
+                if 'retain' in register:
+                    json_messages_retain[register['pub_topic']] = register['retain']
+            else:
+                retain = register.get('retain', False)
+                self._mqtt_client.publish(self.prefix+register['pub_topic'], value, retain=retain)
+
+        # Transmit the queued JSON messages.
+        for topic, message in json_messages.items():
+            m = json.dumps(message, sort_keys=True)
+            self._mqtt_client.publish(self.prefix+topic, m, retain=json_messages_retain[topic])
 
     def _on_connect(self, client, userdata, flags, rc):
         if rc == 0:
@@ -121,6 +140,7 @@ def _on_subscribe(self, client, userdata, mid, granted_qos):
 
     def _on_message(self, client, userdata, msg):
         # print("got a message: {}: {}".format(msg.topic, msg.payload))
+        # TODO Handle json_key writes. https://github.com/tjhowse/modbus4mqtt/issues/23
         topic = msg.topic[len(self.prefix):]
         for register in [register for register in self.registers if 'set_topic' in register]:
             if topic != register['set_topic']:
@@ -147,9 +167,47 @@ def _on_message(self, client, userdata, msg):
                 continue
             self._mb.set_value(register.get('table', 'holding'), register['address'], int(value), register.get('mask', 0xFFFF))
 
+    # This throws ValueError exceptions if the imported registers are invalid
+    @staticmethod
+    def _validate_registers(registers):
+        all_pub_topics = set()
+        duplicate_pub_topics = set()
+        # Key: shared pub_topics, value: list of json_keys
+        duplicate_json_keys = {}
+        # Key: shared pub_topics, value: set of retain values (true/false)
+        retain_setting = {}
+
+        # Look for duplicate pub_topics
+        for register in registers:
+            if register['pub_topic'] in all_pub_topics:
+                duplicate_pub_topics.add(register['pub_topic'])
+                duplicate_json_keys[register['pub_topic']] = []
+                retain_setting[register['pub_topic']] = set()
+            if 'json_key' in register and 'set_topic' in register:
+                raise ValueError("Bad YAML configuration. Register with set_topic '{}' has a json_key specified. This is invalid. See https://github.com/tjhowse/modbus4mqtt/issues/23 for details.".format(register['set_topic']))
+            all_pub_topics.add(register['pub_topic'])
+
+        # Check that all registers with duplicate pub topics have json_keys
+        for register in registers:
+            if register['pub_topic'] in duplicate_pub_topics:
+                if 'json_key' not in register:
+                    raise ValueError("Bad YAML configuration. pub_topic '{}' duplicated across registers without json_key field. Registers that share a pub_topic must also have a unique json_key.".format(register['pub_topic']))
+                if register['json_key'] in duplicate_json_keys[register['pub_topic']]:
+                    raise ValueError("Bad YAML configuration. pub_topic '{}' duplicated across registers with a duplicated json_key field. Registers that share a pub_topic must also have a unique json_key.".format(register['pub_topic']))
+                duplicate_json_keys[register['pub_topic']] += [register['json_key']]
+                if 'retain' in register:
+                    retain_setting[register['pub_topic']].add(register['retain'])
+        # Check that there are no disagreements as to whether this pub_topic should be retained or not.
+        for topic, retain_set in retain_setting.items():
+            if len(retain_set) > 1:
+                raise ValueError("Bad YAML configuration. pub_topic '{}' has conflicting retain settings.".format(topic))
+
     def _load_modbus_config(self, path):
         yaml=YAML(typ='safe')
-        return yaml.load(open(path,'r').read())
+        result = yaml.load(open(path,'r').read())
+        registers = [register for register in result['registers'] if 'pub_topic' in register]
+        mqtt_interface._validate_registers(registers)
+        return result
 
     def loop_forever(self):
         while True:

modbus4mqtt/version.py

@@ -1 +1 @@
-version="0.3.5"
+version="0.4.0"

tests/test_json_key.yaml

@@ -0,0 +1,15 @@
+ip: 192.168.1.90
+registers:
+  - pub_topic: "publish"
+    json_key: "A"
+    address: 1
+    retain: true
+  - pub_topic: "publish"
+    address: 2
+    json_key: "B"
+    value_map:
+      on: 1
+      off: 2
+  - pub_topic: "publish2"
+    address: 3
+    json_key: "A"

tests/test_mqtt.py

@@ -366,5 +366,73 @@ def test_address_offset(self):
 
                 mock_mqtt().publish.assert_any_call(MQTT_TOPIC_PREFIX+'/publish', 2, retain=False)
 
+    def test_json_key(self):
+        # Validating the various json_key rules is among the responsibilities of test_register_validation() below.
+        with patch('paho.mqtt.client.Client') as mock_mqtt:
+            with patch('modbus4mqtt.modbus_interface.modbus_interface') as mock_modbus:
+                mock_modbus().connect.side_effect = self.connect_success
+                mock_modbus().get_value.side_effect = self.read_modbus_register
+
+                m = modbus4mqtt.mqtt_interface('kroopit', 1885, 'brengis', 'pranto', './tests/test_json_key.yaml', MQTT_TOPIC_PREFIX)
+                m.connect()
+
+                self.modbus_tables['holding'][0] = 0
+                self.modbus_tables['holding'][1] = 1
+                self.modbus_tables['holding'][2] = 2
+                self.modbus_tables['holding'][3] = 3
+                m.poll()
+
+                mock_mqtt().publish.assert_any_call(MQTT_TOPIC_PREFIX+'/publish2', '{"A": 3}', retain=False)
+                mock_mqtt().publish.assert_any_call(MQTT_TOPIC_PREFIX+'/publish', '{"A": 1, "B": "off"}', retain=True)
+
+    def test_register_validation(self):
+        valids = [[     # Different json_keys for same topic
+            {'address': 13049, 'json_key': 'a', 'pub_topic': 'ems/EMS_MODE'},
+            {'address': 13050, 'json_key': 'A', 'pub_topic': 'ems/EMS_MODE'},
+            {'address': 13050, 'json_key': 'b', 'pub_topic': 'ems/EMS_MODE'}
+        ],
+        [               # Different topics, duplicate json_key
+            {'address': 13050, 'json_key': 'A', 'pub_topic': 'ems/EMS_MODEA'},
+            {'address': 13050, 'json_key': 'A', 'pub_topic': 'ems/EMS_MODEB'}
+        ],
+        [               # Different topic, no json_key
+            {'address': 13050, 'pub_topic': 'ems/EMS_MODEA'},
+            {'address': 13050, 'json_key': 'A', 'pub_topic': 'ems/EMS_MODEB'}
+        ],
+        [               # Retain specified twice and consistent
+            {'address': 13050, 'json_key': 'A', 'pub_topic': 'ems/EMS_MODE', 'retain': True},
+            {'address': 13050, 'json_key': 'B', 'pub_topic': 'ems/EMS_MODE', 'retain': True}
+        ]]
+        invalids = [[   # Duplicate json_key for a topic
+            {'address': 13050, 'json_key': 'A', 'pub_topic': 'ems/EMS_MODE'},
+            {'address': 13050, 'json_key': 'A', 'pub_topic': 'ems/EMS_MODE'}
+        ],
+        [               # Missing json_key for a register with a duplicated pub_topic
+            {'address': 13049, 'pub_topic': 'ems/EMS_MODE'},
+            {'address': 13050, 'json_key': 'A', 'pub_topic': 'ems/EMS_MODE'}
+        ],
+        [               # Retain specified twice and inconsistent
+            {'address': 13050, 'json_key': 'A', 'pub_topic': 'ems/EMS_MODE', 'retain': True},
+            {'address': 13050, 'json_key': 'B', 'pub_topic': 'ems/EMS_MODE', 'retain': False}
+        ],
+        [               # set_topic and json_key both specified
+            {'address': 13050, 'json_key': 'A', 'pub_topic': 'ems/EMS_MODE', 'set_topic': 'ems/EMS_MODE/set', 'retain': True},
+            {'address': 13050, 'json_key': 'B', 'pub_topic': 'ems/EMS_MODE', 'retain': False}
+        ]]
+        for valid in valids:
+            try:
+                modbus4mqtt.mqtt_interface._validate_registers(valid)
+            except:
+                self.fail("Threw an exception checking a valid register configuration")
+        for invalid in invalids:
+            fail = False
+            try:
+                modbus4mqtt.mqtt_interface._validate_registers(invalid)
+            except:
+                fail = True
+            if not fail:
+                self.fail("Didn't throw an exception checking an invalid register configuration")
+
+
 if __name__ == "__main__":
     unittest.main()