[Tester Wanted] Feature: DPL: support for multiple inverters

[schlimmchen]

I am delighted to share this changeset with all of you: This implements my approach to support multiple inverters by the DPL.

Design

Very shortly (to be explained in the docs in detail):

• Calculate he household consumption
• Try to match it using solar-powered inverters
• Check how much power the battery may provide (nothing (empty), any (full), or limited to solar-charger output (solar-passthrough) and/or battery discharge limit)
• Try to match the remaining consumption (not covered by solar-powered inverters) using battery-powered inverters
• Choosing new limits is based on the difference the DPL must achieve to match the new amount of power requested
• Prefer inverters producing, only shut them down if needed to achieve a particular reduction => subject to debate
• Sort inverters by the amount they can add to a particular increase or reduction of total output power (power diff), such that a minimal amount of inverters needs to receive an update to implement the power diff.
• Manage DPL inverters in a new class, one instance for each inverter selected to be managed.

Status

Still a work in progress, but usable. My setup is running variants of this implementation since early September and from what I see it does what it is supposed to do.

• The WebApp is missing texts/translations.
• The DPL Web UI shall implement input sanitation for newly added or edited inverters.
• The UX when editing a managed inverter needs to improve. Other than general settings for inverters, changes are not applied when closing the modal. Instead, they are submitted together with the DPL settings. My best idea is to show a box with a warning that the settings still need to be submitted once the modal closed and the settings are different.
• Changed DPL settings are not all picked up. There needs to be an async-safe function that applies new settings.
• The WebApp has glitches.
• yarn linting fails, I know why, but I learned just now only, and this will be solved another time.
• Importing settings from older versions, i.e., when upgrading, should work, but I have yet not checked in detail. The glitches I observed when I updated to this version on my productive system have been ironed out.
• Support for solar-powered inverters is there, but the function "tell me how much more this inverter could produce" is a stub: "Sure, I can produce 100W more unless I am already at my configured max limit". @AndreasBoehm the stub is this:

  OpenDTU-OnBattery/src/PowerLimiterInverter.cpp

  Line 267 in 96b0689

  return std::min(getConfiguredMaxPowerWatts() - getCurrentOutputAcWatts(), 100);

• There is no balancing of inverter output. While there is otherwise no need to adjust the inverter limits to match the requested total output, we could instead change the limits of at least two inverters to balance their output. This might by done in the future (this is merely nice-to-have).
• The total inverter power limit setting is available in the web UI and configuration struct, but the value is not yet respected. Only the inverter-specific limits are currently respected.
• Need to check whether a "current limit" of 0 leads to problems, as the inverters report 0W as their current limit after OpenDTU-OnBattery starts and until the information is fetched from the inverter, which can take a while, see the FAQ. These inverters are now not "eligible" and do not participate in achieving the desired output.
• We discussed that it would be a better idea to keep as many inverters on as possible, such that they run with lower temperatures and possibly with higher overall efficiency (which might be false if the limits are very low).
• Under the hood: Remove PowerLimiterInverter::isValid() and integrate into PowerLimiterInverter::create().
• Under the hood: Deal with not knowing the maximum power output without marking the inverter as invalid. Instead, do not consider the inverter for changes until this value was fetched.
• Under the hood: Inverter debug info is missing data: is reachable? is commands enabled? Need to add at least all data which leads to increase or decrease values of 0 (inverter does not participate in achieving a total limit).
• There are TODOs buried in PowerLimiter*.cpp files that need to be addressed.
• Full-Solar-Passthrough Start-Schwellwert and Full-Solar-Passthrough Stop-Schwellwert are initialized to 100, which is outside the input fields limit, preventing to save the DPL settings. See [Tester Wanted] Feature: DPL: support for multiple inverters #1216 (comment).
• Avoid v-show in DPL admin view of web UI.
• Inverters deleted from the DTU entirely should not be listed as managed inverters in the UI.

Preliminary Results

These logs are now outdated, as the implementation progressed since they were recorded.

• Log while toaster was turned on and off
• Log while stove was turned on and off
• Log while oven was turned on and off

Breakfast:

The power meter reading is not very impressive 😞 I hope that it is okay since the issue of stoves using power in intervals is well known. The resolution of the power meter graph is only 10s. I don't know why, it should be 1s, I never bothered to check.

Testing

As stated above, this is usable, at least for systems with multiple battery-powered inverters, and I deployed it to my productive system with one HMS-2000 and one HM-1500. I would be very happy to receive your feedback on this. Firmware can be downloaded from the respective PR build run.

Please do not open issues but answer to this PR when giving feedback.

Related

Closes #230.
Closes #1032.
Closes #1071.

[DonJohnLong]

Läuft seit heute im Produktivsystem meines Vaters mit 2x hm-600, aufgefallen ist mir auf die schnelle, das sich die regel gschwindigkeit der DTU addiert, was aber irgedwie logisch ist. 1 sek = 2 sek pro wechselrichter.

Ansonsten läuft es bestechend gut, bei leistungen ab 300w sind die wechselrichter nahe bei einander 155w/145w o.ä.

Bei kleineren leistungen hab ich auch schon 45w/93w geshen. Das eingestellte Limit wird jedoch tadellos getroffen.

Ansonsten ist mir soweit nichts negatives aufgefallen 😁

Da das Sytem leider nicht bei mir steht, kann ich leider nur alle paar Tage draufschauen. Laut Shelly ist der Verbrauch aber da wo er sein sollte.

Vielen Dank @schlimmchen für deinen betriebenen Aufwand. Funktioniert grossartig bis jetzt 👍

Soll ich bei Gelegenheit irgend etwas speziell testen? Lg

[schlimmchen]

@DonJohnLong Vielen Dank fürs mutige Testen und deine Rückmeldung!

aufgefallen ist mir auf die schnelle, das sich die regel gschwindigkeit der DTU addiert, was aber irgedwie logisch ist. 1 sek = 2 sek pro wechselrichter.

Naja, nachregeln sollte eigentlich so schnell sein wie zuvor, denn es sollte in aller Regel ja weiterhin nur ein Inverter ein Update erhalten müssen, um den neuen Haushaltsverbrauch zu kontern. Wenn du allerdings vorher nur einen WR an je einer OpenDTU hattest, dann dauert es natürlich etwas länger als zuvor. Da möchte ich noch dran schrauben, aber bisher sind wir da auf 1s zwischen den WR und 1s Pause zwischen den Runden beschränkt.

Dass die Leistungen der Inverter nahe beieinander sind ist nur ein Bonbon und kann nicht garantiert werden. Es kommt drauf an, wie der Haushaltsverbrauch schwankt und wie die Hysterese eingestellt ist, etc. Aber in der Tat, wenn die Hysterese klein ist und die Inverter gleiche maximale Ausgangsleistung haben und die PowerMeter Schwankungen eher klein sind, aber größer als die Hysterese, dann nähern sich die Leistungen über die Zeit an.

Soll ich bei Gelegenheit irgend etwas speziell testen? Lg

Lieb, dass du fragst, aber ich finds schon total super, dass du hilfst zu testen und Rückmeldung gibst, ob dir etwas negativ auffällt, das ist erstmal mehr als ausreichend 💪

[ingrimsch]

installed the build today and apart from not seeing the power limits in the webapp it seems to do its job so far. Thank you very much for implementing this, I will monitor the behavior closely and report if I stumble upon anything out of the ordinary👍

edit: small correction - I can see the power limits now (was "0" before for both already working inverters)

[schlimmchen]

I will try to provide the solar powered inverter implementation until sunday :)

No worries and no rush. Let's do a maintenance release soon to fix issues users reported, then take a couple of weeks to finalize and merge this as well as other features/PRs (which there are quite a lot all of a sudden).

Even if i change it to 52,1 it will return back to the shown value.

Yes. This drives drove me nuts. See #1225.

[ingrimsch]

Quick feedback: looks good to me so far

Rainy day completely ruins the otherwise awesome stats I normally achieve using this excellent project xD

PS: dont trust todays values, "heute eingespeist" only counts one inverter, I forgot to reboot-proof the 2nd one

[schlimmchen]

Quick feedback: looks good to me so far

Fancy dashboard.

So what you haven't observed, obviously, is that one inverter might be off for a longer period of time simply because it is not needed. This night, my inverters were shut off as the battery was depleted. After solar power was available again, one started producing power (the bigger one with 2kW output, as that is sorted to the front of the queue). Since then, there was no situation where the first one produced at least 500W such that the second one (1.5kW output) would have been sorted to the front of the list as more power was needed.

I don't like that. I anticipated that this could happen while writing the code, and I think I am going to sort inverters in shutdown to the front of the list when an increase is needed. This still requires a jump of at least the lower power limit of that inverter such that it will turn on, but I guess that happens rather frequently, even if it's not a busy day.

Also I want to find a way to decide when to sort the list by a different metric, i.e., such that the output power of both inverters align (in absolute output or relative output?!). Probably it's a good idea to so that every time any inverter can take the update to achieve the new total output.

[AndreasBoehm]

I thought that we want to keep the number of active inverters low to prevent that we are running them on low limits which results in the output not being what we set as the limit (HMS-2000 below 15%) and because every extra MPPT brings more voltage variance in to the system. Please correct me if i misunderstood something.

[schlimmchen]

Hm. Good point. Do you have suggestions on how to implement it? Right now I did the implementation with the opposite assumption in mind. In particular, an inverter will not be turned off unless it has to be. I need to think about that a little.

And if we do it like this, I really think that there needs to be some kind of load balancing to spread the aging across the inverters somewhat evenly. Maybe we keep the inverter running which, proportional to its max output, has produced less energy? However, it would also be very nice if we switch inverters based on their temperatures, so that we allow them to run cooler?

You opened a can of worms 😉

[zaziki23]

I would like to test this once i am back home and can flash my devices.
I currently run two on battery instances to control two HMS-2000 inverters.
As from my experience with two inverters in general so far, i would definitly try to use only as much inverters at the same time as needed. Which should also have the benefit of more speed as long as only one inverter is active, which should be most of the time i guess. HMS Devices also come down to a nice stand by consumption of 1,2W only if they are not needed.

[schlimmchen]

I would like to test this once i am back home and can flash my devices.

Nice!

Which should also have the benefit of more speed

No, why do you think so? The answer is no because the current implementation sorts the inverters by the amount they can increase or decrease their output, and the one with the biggest difference is at the front This means only one inverter is updated at a time.

i would definitly try to use only as much inverters at the same time as needed.

Are there other reasons from your perspective other than the speed argument, which I think is not a concern?

[zaziki23]

Efficiency and Simplicity

If one inverter is enough to cover the household, it should be more efficient as two running at low load.
If two are running, you still need to get power ratings for both of them. If one is disabled this can be skipped.

[schlimmchen]

thank you, I can see your points, very good arguments! let me skip waiting for inverter data if the inverter is currently not in use to speed up the DPL loop, possibly considerably. and let me change the "algorithm" to use as little inverters as possible. I still need to think about it, though. any ideas from you guys?

[AndreasBoehm]

To be honest i dont think its as easy to say that the minimum number of inverters is the best way to go.

If we are close to the limit of the first inverter we will have issues to react to increased power requirements quickly. Because it takes before production starts and because we need to reach the minimum limit of the second inverter.

what about setting an upper limit of, i dont know, maybe 75% and then we start a second inverter to be able to quickly increase the output?

[spcqike]

I also think it’s better if all inverters run in a low but stable level.

I’m not to sure if efficiency is a real problem. A inverter at 70% load may be more efficient than on 15%. But losses across the cables will be lower when using two inverters, as you reduce the current for each pair of cable and so you basically double the wire diameter.

And I think two inverters at 30% are cooler than one at 60%. Low temperatures are better for the hardware in the long run

I don’t know if the inverters need to reach their lowest limit where they can hold the desired power (like ~15% for the HMS-2000) or if it’s enough when the first one runs stable (10-15W per MPPT, so no communication errors but also not reaching the target limit) and the second produces the missing power to hold your overall power target.
case A would be tricky as you need 300W for a HMS-2000 to hold its target limit. But B would be more complex to calculate or slower to settle in, somehow.

Because it takes before production starts

is that true? I thought it doesn’t matter if the inverter is stopped or does produce, as long as it’s not disconnected from the grid or restarts. The first start takes time for grid Synchronisation, but after that it is quite fast, I thought.

[schlimmchen]

I read this and I am not confident that we can reach a common ground in this. Maybe we don't need to. It is understandable that ine strategy does not fit all use cases or setups.

Using different sorting of the inverters, I think we can quite easily implement very different strategies.

Towards the extremes, all inverters should be on or off in any case. So I dont' think that changing the implementation design to iterate the sorted inverters and call the respective increase or decrease method is required.

It is also easy to drop an inverter from the sorted list, if the sorting strategy decides that it does not want an inverter to be considered in the current round. The algorithm could also cap the increase or decrease for the current round. Adding methods for that to the DPLinverter class is straightforward.

The implementation of these strategies can go into their own class each, implementing a particular interface, which is simple to read and maintain.

I had this in mind before, that's why I was very happy with the approach to sort inverters, as I realized that this could easily be adapted. We can make this even more flexible as described above.

[schlimmchen]

Fixed ✅

@AndreasBoehm There are two problems here:

1. This kind of overflow (8 - 10 in this case) must not happen, of course. Such an error shall be handled gracefully and I will implement a fix.
2. The reason you triggered this bug is because your lower limit is far to little. 20W may have worked for an HM-600, but an HMS-2000 with 4 inputs shall have a lower power mit of at least 60W (maybe 40W works as well). The logs tells you ecatly that: The limit is already at 8W, but the inverter produces 47W. The inverters don't do well with such low limits.

Done ✅

Also, I continue to get headaches thinking about how to tweak the methods that calculate the possible reduction/increase as well as those applying them, because they need to deal with solar-powered inverters as welll as battery-powered inverters. I think I want to split PowerLimiterInverter into PowerLimiterSolarInverter and PowerLimiterBatteryInverter (with PowerLimiterInverter being an abstract base class). Unfortunately, we then have to organize the instances using pointers to the base class... However, since I already use a deque, those objects already "float around" in the heap.

[schlimmchen]

It does not make sense that the limit should be 8W, maybe thats also a problem of the new DPL implementation?

Yes, of course. I did not want to suggest that you are at fault, I just wanted to point out how you managed to trigger this issue. Good job 😉

However, I still maintain that your lower limit is too low, given the joint experience we shared in this project. Did you not notice that the inverter is shutting itself off at these low limits? Is the DPL restarting the inverter because of that (check the logs). Maybe your specific inverter is not prone to shutting itself down due to oscillations...

[schlimmchen]

I have observed the oscillations until the inverter shuts off with my HM-1500. My HMS-2000 never had very low limits set. Mine is week 31 of 2023 and has firmware 1.0.16. My das has one such model as well, and one the same as yours. Maybe I will torture them with very low limits and see how they behave -- some day.

[schlimmchen]

Fixed ✅

There is another stupid mistake, another underflow: It occurs if some non-DPL-managed inverter or completely other power source makes the power meter value go "too negative". The DPL will consider the DPL-manged inverters' output, but if the result is still negative, we (I...) interpret this value as an uint16_t, which is garbage, of course 🤦‍♂️ If we already feed into the grid, the power request to the inverters shall be zero.

Log when requested power value underflows

12:48:55.722 > [DPL::loop] ******************* ENTER **********************
12:48:55.725 > [DPL::loop] battery interface enabled, SoC: 5 %, StartTH: 45 %, StopTH: 40 %, SoC age: 0 s, ignore: yes
12:48:55.747 > [DPL::getBatteryVoltage] BMS: 51.98 V, MPPT: 51.89 V, inverter 116183125666: 51.80 V, returning: 51.98V
12:48:55.749 > [DPL::loop] dcVoltage: 51.98 V, loadCorrectedVoltage: 52.44 V, StartTH: 52.00 V, StopTH: 51.00 V
12:48:55.749 > [DPL::loop] StartTH reached: yes, StopTH reached: no, SolarPT enabled, use at night: no
12:48:55.749 > [DPL::calcHouseholdConsumption] target consumption: 20 W, base load: 400 W
12:48:55.751 > [DPL::calcHouseholdConsumption] power meter value: -2940.0 W, power meter valid: yes
12:48:55.753 > [DPL::calcHouseholdConsumption] inverter 116183125666 is behind power meter producing 1522 W
12:48:55.756 > [DPL::updateInverterLimits] requested: 64098 W, producing: 0 W using 0 solar-powered inverters, diff: 64098 W, hysteresis: 10 W
12:48:55.759 > [DPL::calcBatteryAllowance] power requested: 64098 W
12:48:55.762 > [DPL::updateInverterLimits] requested: 64098 W, producing: 1522 W using 1 battery-powered inverters, diff: 62576 W, hysteresis: 10 W
12:48:55.766 > [DPL::updateInverterLimits] will cover 1522 W using battery-powered inverters
12:48:55.770 > [DPL inverter 116183125666]: debug info:
12:48:55.772 >     solar powered: no
12:48:55.774 >     output capability: 1500 W
12:48:55.777 >     upper power limit: 1500 W
12:48:55.779 >     lower power limit: 50 W
12:48:55.784 >     producing: yes
12:48:55.788 >     current output: 1522 W
12:48:55.793 >     current limit: 1500 W
12:48:55.795 >     max reduction: 1472 W (online), 1522 W (standby)
12:48:55.798 >     max increase: 0 W
12:48:55.803 >     expected (new) output: 1522
12:48:55.808 >     update timeouts: 0
12:48:55.810 > [DPL::loop] consumption: 64098 W, solar inverters output: 0 W, battery allowance: 64098 W, battery inverters output: 1522 W

Kudos to @AndreasBoehm for showing us that collapsible sections are available in Github comments. ❤️

[schlimmchen]

Fixed ✅

Aaaand another stupid mistake... This is definitely not as robust as I had hoped 😞 At least not yet...

My HM-1500 is not turned off even though the battery stop threshold is reached. That's because it has a limit of 49W set (guess that's a rounding error), whereas the lower power limit is 50W, so the PowerLimitInverter class assumes the inverter cannot reduce the power output...

Log of inverter not being shut down

20:37:21.021 > [DPL::loop] ******************* ENTER **********************
20:37:21.026 > [DPL::loop] battery interface enabled, SoC: 0 %, StartTH: 45 %, StopTH: 40 %, SoC age: 1 s, ignore: yes
20:37:21.029 > [DPL::getBatteryVoltage] BMS: 51.09 V, MPPT: 51.17 V, inverter 116493100759: 51.30 V, returning: 51.09V
20:37:21.032 > [DPL::loop] dcVoltage: 51.09 V, loadCorrectedVoltage: 51.10 V, StartTH: 52.00 V, StopTH: 51.00 V
20:37:21.034 > [DPL::loop] StartTH reached: no, StopTH reached: no, SolarPT enabled, use at night: no
20:37:21.037 > [DPL::calcHouseholdConsumption] target consumption: 20 W, base load: 400 W
20:37:21.039 > [DPL::calcHouseholdConsumption] power meter value: 471.0 W, power meter valid: yes
20:37:21.046 > [DPL::calcHouseholdConsumption] inverter 116183125666 is behind power meter producing 41 W
20:37:21.049 > [DPL::calcHouseholdConsumption] inverter 116493100759 is behind power meter producing 0 W
20:37:21.051 > [DPL::updateInverterLimits] requested: 492 W, producing: 0 W using 0 solar-powered inverters, diff: 492 W, hysteresis: 10 W
20:37:21.054 > [DPL::calcBatteryAllowance] power requested: 492 W
20:37:21.058 > [DPL::calcBatteryAllowance] limited to solar power: 0 W
20:37:21.063 > [DPL::updateInverterLimits] requested: 0 W, producing: 41 W using 2 battery-powered inverters, diff: -41 W, hysteresis: 10 W
20:37:21.066 > [DPL::updateInverterLimits] will cover 41 W using battery-powered inverters
20:37:21.068 > [DPL inverter 116183125666]: debug info:
20:37:21.071 >     solar powered: no
20:37:21.074 >     output capability: 1500 W
20:37:21.076 >     upper power limit: 1500 W
20:37:21.079 >     lower power limit: 50 W
20:37:21.081 >     producing: yes
20:37:21.084 >     current output: 41 W
20:37:21.088 >     current limit: 49 W
20:37:21.090 >     max reduction: 0 W (online), 0 W (standby)
20:37:21.093 >     max increase: 1451 W
20:37:21.096 >     expected (new) output: 41
20:37:21.099 >     update timeouts: 0
20:37:21.102 > [DPL inverter 116493100759]: debug info:
20:37:21.105 >     solar powered: no
20:37:21.107 >     output capability: 2000 W
20:37:21.110 >     upper power limit: 2000 W
20:37:21.114 >     lower power limit: 60 W
20:37:21.118 >     producing: no
20:37:21.121 >     current output: 0 W
20:37:21.127 >     current limit: 70 W
20:37:21.130 >     max reduction: 0 W (online), 0 W (standby)
20:37:21.136 >     max increase: 2000 W
20:37:21.140 >     expected (new) output: 0
20:37:21.144 >     update timeouts: 0
20:37:21.147 > [DPL::loop] consumption: 492 W, solar inverters output: 0 W, battery allowance: 0 W, battery inverters output: 41 W

[schlimmchen]

No significant progress, unfortunately, but rebased onto current development branch. New firmware is linked in the PR description.

[rradkov]

Is there any need for additional testers?
Currently I have two HMS inverters - 2000-4T and 1000-2T and shelly EM smart meter.

[schlimmchen]

@rradkov It makes me happy to see that more and more users use this preview-firmware and that they can achieve the desired result of governing more than just one inverter. So I am glad if you join us. However, there is no real need to do so for the sake of testing. I am actually quite sure that this version of the feature is quire usable, as I do as well on my productive system. Notice the open tasks in the PR description. Upgrade to this firmware if you feel comfortable doing so.