Add TCP source for lightweight, extended data gathering

[MarijnS95]

Certain Omnik inverters (supposedly with a specific serial number, see sources below) reply with its statistics as binary data when a "magic" packet is sent to them on port 8899 over TCP. This data is more lightweight than an HTML page or JavaScript source with values embedded, and provides more statistics then either similar to what can be found on the LCD display.

The data format and meaning has been extracted from various Python/C sources (see links below), with some of my own "work" to simplify checksum calculation and "the reverse hexadecimal serial number" is nothing more than its little-endian byte representation. Response data is parsed at once with a ctypes.BigEndianStructure instead of unpacking individual ranges of bytes through struct.unpack. Only minimal data massaging is needed to convert the struct to useful information.

It provides the following extra fields:

• Inverter temperature;
• Voltage and current for the DC input strings (up to 3);
• Voltage, current, frequency and power for all AC outputs (also up to 3);
• Total number of runtime hours (not sure if this is just the Omnik being on, or the time it delivers >0W of power back to the network).

No "device" (network module) data is found in any of the unparsed bits.

I have also not been able to extract solar_rated_power from the data. Comparing to http://www.mb200d.nl/wordpress/2015/11/omniksol-4k-tl-wifi-kit/#more-590 it seems the second byte in the "magic" is 0x81 on a 4k-TL instead of the 0x7d on our 3k-TL. It is just speculation thus far, but this might be an integer identifier of the hardware type? (To be confirmed by the community, I guess). Otherwise padding0 contains a lot of non-zero data too.

Perhaps anyone is interested in dumping the remaining data and seeing if they can uncover their meaning. We could help this by printing the padding fields if they're not what we expect (mostly zero).

Sources:

1. https://github.com/jbouwh/omnikdatalogger/blob/dev/apps/omnikdatalogger/omnik/InverterMsg.py
2. http://www.mb200d.nl/wordpress/2015/11/omniksol-4k-tl-wifi-kit/#more-590
3. https://github.com/Woutrrr/Omnik-Data-Logger
4. https://github.com/t3kpunk/Omniksol-PV-Logger

[klaasnicolaas]

Thank you for your contribution,

It's a pretty big change and is partly beyond my cap. Despite that, I think it would be nice to see what possibilities there are with the various inverters. Could you see if you can pass the linting tests?

[MarijnS95]

It's a pretty big change

I can try to split this up (in separate commits and/or PRs) if that helps? Don't immediately see a clear boundary though.

and is partly beyond my cap.

Should go without saying that I'll be utilizing this python package (through https://github.com/robbinjanssen/home-assistant-omnik-inverter once I submit related changes matching this PR there), and I'll be sure to stick around this repository to help out with review and clarification regarding the TCP connection "backend". I'm subscribed to notifications but feel free to @ me if anything shows up!

That's also important regarding the unknown fields and potential changes in the "magic", as detailed in the PR description.

Despite that, I think it would be nice to see what possibilities there are with the various inverters.

Yeah, I hope https://github.com/robbinjanssen/home-assistant-omnik-inverter is open to add logging for all the new fields. Besides temperature most is just a gimmick (though being able to see string balancing could be relevant?), but it'd be interesting to see if one of the unknown fields (or "magic") change across various inverters.

Could you see if you can pass the linting tests?

Thanks, I'll of course make sure this adheres to all the checks! Bit Rusty (:smirk: I write Rust now daily) on the Python side, I hope the code is at least in an acceptable state as it's quite tricky to find the right - concise - ways to write things in Python again. I managed to replicate most failing tests locally, will update this PR soon.

[MarijnS95]

Another two things that came up:

• Perhaps we can support multiple source types;
  • Use auth-less js to extract the serial number automatically, before attempting to use tcp going forward;
  • Send a js request to gather Device information in addition to the tcp request with extended statistics.
• Maybe more Python Omnik implementations are willing to switch over to this package instead of maintaining a copy of "the TCP code" themselves.

[klaasnicolaas]

Could you DM me on one of my socials? I would like to invite you to our slack server, where both me and Robbin are in for the whole omnik development (python package and integration). Also more convenient for the quick communication 😉

[MarijnS95]

@klaasnicolaas I think you need just my email for that, which you can pull straight from the commit body :)

[MarijnS95]

Since all of these are @classmethod anyway, perhaps these functions should just move into the module directly? Seems like an unnecessary indirection in hindsight.

[codecov]

Codecov Report

Merging #134 (bf54d05) into main (548edbb) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##              main      #134    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files            4         6     +2     
  Lines          182       325   +143     
  Branches        38        65    +27     
==========================================
+ Hits           182       325   +143     

Impacted Files	Coverage Δ
omnikinverter/const.py	100.00% <100.00%> (ø)
omnikinverter/exceptions.py	100.00% <100.00%> (ø)
omnikinverter/models.py	100.00% <100.00%> (ø)
omnikinverter/omnikinverter.py	100.00% <100.00%> (ø)
omnikinverter/tcp.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 548edbb...bf54d05. Read the comment docs.

[github-actions]

There hasn't been any activity on this pull request recently. This pull request has been automatically marked as stale because of that and will be closed if no further activity occurs within 7 days. Thank you for your contributions.

[MarijnS95]

Not entirely sure what's going on here. pylint wants me to test:

python-omnikinverter/omnikinverter/omnikinverter.py

Lines 151 to 154 in 935ab48

	except Exception as exception:
	raise OmnikInverterConnectionError(
	"Failed to communicate with the Omnik Inverter device over TCP"
	) from exception

Yet I can't check for that specific exception here, even though it is clearly visible in the logs. Instead, I have to check for the much less specific Exception("Connection broken") here that is raised by asynctest for us.

[MarijnS95]

I found what is going on, and the details are in comparing raise ... from ... re-wrapping (with The above exception was the direct cause of the following exception:) working in test_connection_failed(), but not in this test_connection_broken(self) which shows During handling of the above exception, another exception occurred:.

Turns out the exception is ALSO raised in the finally: block through the .wait_closed() call, directly while we were handling OSError and calling raise OmnikInverterConnectionError(...) from ..., as per the During handling of the above exception, another exception occurred:.