unified shv log format

SHV remote log

getLog

getLog({"since": dt1|"last", "until": dt2|null, "withSnapshot": true|false, "withPathsDict": true|false, "recordCountLimit": 1000 })

parameters:

• since - default: null
  • dt1 - time of first returned record
  • null - time of first record in history available
  • "last" - obviously used together with withSnapshot == true, time of the most recent record in history log is taken a since value.
• until - default: null
  • dt2 - time of last returned record exclusively, records with TS >= dt2 will not be part of response
  • null - all records or until recordCountLimit is hit
• withSnapshot - default false, return snapshot of all values in time == since, all the snapshot entries will have time == since
• withPathsDict - default true, return pathsDict in metadata. PathsDict maps every entry path to unique int number. This can save some space in returned log.
• recordCountLimit - default 1000, maximum number of records returned, client is free to return less than recordCountLimit requested.

Unified SHV log format

Unified log format is used when it is retrieved from any SHV device.

Example:

<
	"dateTime":d"2022-12-16T20:57:44.908Z",
	"logParams":{
		"recordCountLimit":10000,
		"since":d"2022-12-15T20:57:40.297Z",
		"until":d"2022-12-16T21:57:40.297Z",
		"withPathsDict":true,
		"withSnapshot":false,
		"withTypeInfo":false
	},
	"logVersion":2,
	"pathsDict":i{
		0:"shv/system/1/voltage600",
		1:"shv/heating/1/airTemp",
		2:"shv/vet/1/vehicleDetected",
		3:"shv/vet/1/status",
		4:"shv/vet/1/status/direction",
		5:"shv/tc/1/status/occupied",
		6:"shv/tc/1/status",
		7:"shv/system/1/status/blocked",
		8:"shv/system/1/status",
		9:"shv/ppi/1/status/blocked",
		10:"shv/ppi/1/status",
		11:"shv/tc/2/status/occupied",
		12:"shv/tc/2/status",
	},
	"recordCount":4187,
	"recordCountLimit":10000,
	"recordCountLimitHit":false,
	"since":d"2022-12-15T20:57:40.297Z",
	"until":d"2022-12-16T21:57:40.297Z",
	"withPathsDict":true,
	"withSnapshot":false
>[
	[d"2022-12-15T21:01:09.990Z", 0, 651., null, null, 0u, null],
	[d"2022-12-15T21:02:03.190Z", 1, -5., null, null, 0u, null],
	[
		d"2022-12-15T21:04:01.530Z",
		2,
		{"transceiverPosition":1, "vehicleId":3023, "vehiclePosition":1},
		null,
		null,
		1u,
		null
	],
	[d"2022-12-15T21:04:01.530Z", 3, 0, null, null, 1u, null],
	[d"2022-12-15T21:04:01.530Z", 4, 0, null, null, 0u, null],
	[
		d"2022-12-15T21:04:01.530Z",
		2,
		{"transceiverPosition":1, "vehicleId":3023, "vehiclePosition":1},
		null,
		null,
		1u,
		null
	],
	[d"2022-12-15T21:04:03.860Z", 5, true, null, null, 0u, null],
	[d"2022-12-15T21:04:03.860Z", 6, 1, null, null, 0u, null],
	[d"2022-12-15T21:04:03.860Z", 7, true, null, null, 0u, null],
	[d"2022-12-15T21:04:03.860Z", 8, 2, null, null, 0u, null],
	[d"2022-12-15T21:04:03.960Z", 9, true, null, null, 0u, null],
	[d"2022-12-15T21:04:03.960Z", 10, 18, null, null, 0u, null],
	[d"2022-12-15T21:04:06.060Z", 11, true, null, null, 0u, null],
	[d"2022-12-15T21:04:06.060Z", 12, 1, null, null, 0u, null],
	[d"2022-12-15T21:04:10.860Z", 5, false, null, null, 0u, null],
	[d"2022-12-15T21:04:10.860Z", 6, 0, null, null, 0u, null],
	[d"2022-12-15T21:04:12.760Z", 11, false, null, null, 0u, null],
	[d"2022-12-15T21:04:12.760Z", 12, 0, null, null, 0u, null],
	[d"2022-12-15T21:04:12.760Z", 7, false, null, null, 0u, null],
	[d"2022-12-15T21:04:12.760Z", 8, 0, null, null, 0u, null],
	[d"2022-12-15T21:04:12.860Z", 9, false, null, null, 0u, null],
]

Device log

One possible device log format implementation to support the Unified SHV log format in simple way

• Devices are logging to the text file
• Every entry is one line terminated by \n
• Fields in line are separated by \t
• Minimal fields are:
  1. timestamp - monotonic timestamp with format yyyy-mm-ddThh:MM:ss.zzz
  2. orig_timestamp - timestamp MUST be monotonic, if the device tries to log with timestamp lower than the last record one, then timestamp is adjusted to be monotonic and original timestamp is saved as orig_timestamp. This information is necessary to reconstruct timestamp in rows where device system time wasn't in sync with NTP
  3. path - string
  4. value - Cpon coded logged value for path

Notes:

1. It is a device responsibility to ensure monotonic timestamp. When device boots, it have to read last log entry and remember recently logged timestamp. If device time after reboot is lover than recent TS, it have to log new events with the recent timestamp. Even if new timestamps will not be correct, they can be reconstructed later on, when device will connect to internet and sync system time with NTP, using uptime value.
2. It is necessary to keep timestamp monotonic even if it is not correct to enable binary search in logs and to generate the Unified log format easily.
3. \n and \t are good line and field separators, since they never appear in Cpon packed values
4. One option to implement device logs was to use SQLite. Simple file was chosen according to Do the simplest thing that could possibly work rule from following reasons:
5. Append line to file is really cheep and fast in almost every possible OS
6. To ensure DB consistency even when device is rebooted suddenly, Sqlite will create rollback journal on SD card https://www.sqlite.org/tempfiles.html , this can produce undesirable write cycles on EEPROM.
7. restore corrupted text log file is pretty easy task comparing to do the same thing with Sqlite
8. Open Sqlite with every log entry can spend some system resources to read table structures and init SQL library. App can keep Sqlite connected all its lifetime to come over this.

log file example

2022-01-11T15:38:50.104Z--->123456--->status--->2u--->52345u--->chng
2022-01-11T18:51:37.987Z--->1970-01-01T00:01:37.987Z--->status--->1u--->12345u--->fastchng

Columns:

1. monotonic timestamp
2. original timestamp (if monotonicity is broken)
3. shv/path
4. value
5. short-time - uint16_t number of miliseconds reported by device without RTC
6. domain - values can be grouped to domains

• chng value change, it is used primarily for digital slow signals
• fastchng - analog values changing all the time