[Request] Drop support for devices with 4MB flash (?) #935
Comments
|
Copying my own comment from #923:
It is my personal interest and motivation to help make this project fun to use for everybody. Even though a voice in my head says "screw it, let's go for 8MB+ devices in the future", another voice goes "don't risk people becoming frustrated with this project because we decided to drop supporting their setup". @swingstate idea to plan this in advance could mitigate some frustration. |
|
We could introduce a Long-Term Support (LTS) version of ODTUoB, where only bug fixes would be provided within the limitations of the 4MB and in a couple of years 8MB flash. The LTS support for the 4MB hardware could last until the end of 2026, similar to how Ubuntu is managed. Drawback of such approach would be that this might require maintaining two development streams. One LTS and one Main which is more feature rich. Each release (a number of builds could count to the same release) would specify a minimum hardware requirement. With clear communication and planning, this approach should help reduce frustration and allow enhancing this project. |
|
Another possibility could be to provide different builds like tasmota does. One for NRF, one for CMT, one with oled or without, with ve.direct or without,…. Depending on what parts need much diskspace. As for me, right now I only use a CMT and solar only. No battery, no Victron, no BMS. All this code and libraries could be kept out in my case. Or maybe one could also use the custom „define“ tags like Tasmota does. So everyone could build a firmware as needed. Do we have an overview of which part needs how much space? |
Personally I use Victron MPPT and CANbus for my test systems thus, (for example): Would removing JKS and Huawei resolve to (significant) less memory used?
Assuming that the price difference between ESP32.4Mb and ESP32.8Mb, will be similar to the price difference between ESP8266 ans ESP32, I would dare say that, within 12 months the majority hardware will be ESP32.8Mb So, as swingstate pointed out, I believe that supporting a 4Mb version during 12 months will be more than enough. PS. and as usual, beaten to it by @spcqike 🙄 😆 😆 😆 |
|
Any change to improve OTA? |
|
I am taking a veeeeery wild guess but i assume the ratio of users on 8Mb flash is small, very small. I'd say it's good to start spreading the idea, but to cut the support now would be very noisy. My 2 cents, obviously. P.S. owner of three 4MB ESPs lol |
|
We dont have data, but since the OpenDTU FUSION board has an ESP32-S3 with 8MB (or even 16?), I think that the relative amount is therefore not "very small". I also think that nearly all users with 8MB or more use an OpenDTU FUSION board. swingstate's board is (as far as I can tell) designed to habe 16MB. I really dont like the idea of adding config switches to the firmware to disable functionality. The overhead, especially when maintaining, is a real headache. The amount of different firmware versions will explode very fast, and there will be a at least one that cannot compile for different reasons for most of the time. The costs of the board do not concern me. If we are going for 8MB flash as a requirement, those few bucks should be in everybody's pocket. Why do we have a second firmware partition in the first place? As far as I can tell it is there to boot from in case the other one is corrupted? That is something that I think is luxurious. If corruption occurs, which certainly happens very seldom, we could require users to re-initialize using the bootloader and USB instead. What do you think about that? This will be a breaking change, of course. Most ideas are a breaking change. |
good point. Do you think it would be possible, to have something like a minimalistic second firmware, that only connects to wifi (or starts AP) and let you re-upload / OTA to the desired firmware? something like "tasmota-minimal", which is only 262Kb. i think, some users may have trouble hardware-flashing their boards, which they bought online. so a backup firmware makes sense, just in case. but maybe it mustn't be a complete copy of the firmware but just a minimal one that allows another OTA attempt. |
|
Yes, agreed. However, I have little hope that this minimal Firmware will be indeed very small. And again: Maintaining it and building it is probably a headache. And it needs a minimal WebApp as well, another headache. I like this idea as well, but it's a pain. And will this minimal firmware ever be updated? If not, when does it stop working in some setups because of newer technology or WLAN encryption and such. The main firmware will be updated and stay compatible. The minimal firmware is left as is, and when the day comes and the flash process was not successful, the minimal firmware does start, but cannot connect to the WiFi? Lame.
Notice that changing the partition layout will require flashing the factory firmware bin using the USB interface in any case. If we think we need to avoid this, we are in trouble, as touching the partition layout is suddenly not an option. Let me use this occasion to introduce a new feature. It's not "officially announced" yet, but @helgeerbe and I gave approval and it is online. However, I did not even test it yet. https://solar.metacontrol.eu/opendtu-onbattery-webflasher/ Given that this web flasher is available, there is very little excuse to not update an existing ESP32. Do you agree? |
|
Agree. Great to see this web flasher, one less headache. |
|
out of curiosity tried to find such a board (8 MB) and not successful. any links, preferably international? |
|
As I also asked... Please, one AMAZON Link and one from Asia, if possible. |
Unfortunately not international: |
|
If one wants to cancel 4Mb in the long run, does it make sense to use a specific module with 8Mb or more? Like there is N8, N8R2, N8R8, N16R8 and maybe more. Where Nx is the flash and Rx the PSRAM. Is it advised to use N8R8 with 8Mb each or is N8R2 (or even N8) ok to use? |
|
Thanks swingstate! These are the numbers I was loooking for! @schlimmchen if the prices are at this level already, why not go straight for the 16MB model?
Oh, I see something interesting: 3 x UART Schnittstellen |
I'd second that, since almost all popular ESP32 devboards all only feature 4MB (without even mentioning it).
Times three here 🤣 Which I would replace if it was the only way of keeping up with recent versions, but it would make my heart bleed rendering these capable little workhorses obsolete. Also, I'm certain the pinout will change again and I have to resolder everything 🎉
If you don't like waiting, there are several offers shipped by Amazon for around 8€ per piece, search term [esp32-s3 n16r8].
From the very beginning, I was really amazed by the level of refinement that went into this (and, of course the original openDTU) project, which puts a lot of commercially developed products to shame. On the other hand, the main target group are (and probably always will be) tech-savvy people, which should be either able to help themselves, or to search for solution online. Using a wired connection for every update would be kind of a bummer, but for recovery purposes, it's totally justified, if not industry standard.
A thought I just had was the usage of a modular package system, as implemented by other projects with similar problems, like OpenWrt. I'm not even sure if such thing is possible on the ESP platform, and that this is an absolutely humongous task that also introduces many other challenges, but it could keep the overall flexibility, future development and hardware limitations in some sort of balance. |
|
@schlimmchen, do you have any other thoughts? What do you think about the idea of freezing the current development state (the timing of which is yet to be determined) that is compatible with the 4MB boards? This frozen state would only receive maintenance efforts on a best-effort basis with lower priority, and no new features would be added. Instead, the focus would shift to the 8 or 16MB boards. We would communicate this 'breaking change' 3 to 6 months in advance." |
|
Is this only about the amount of flash, or is the availability of PSRAM also a reason? It would be good to communicate this from the beginning, when the 4MB support is dropped. Whether the baseline and foreseeable future also makes use of the PSRAM or not. There are 8 and 16MB flash versions with 0 and 0,2 and 8MB of PSRAM. Which one is the preferred baseline?
The typical cost of a N16R8 (that is 16MB of flash, 8MB of PSRAM) is about 6€ at aliexpress. (note using bank switching more than 4MB of PSRAM can be used, that's the himem feature). So without understand all the software issues, wouldn't it be better to go for a "non-zero" PSRAM baseline S3 model? |
Interim ConclusionFlashThe consensus actually seems to be to drop support for devices with 4MB flash. This must be announced beforehand. Giving a time-frame, however, is tricky. Let's say we manage to keep the 4MB flash devices alive for now, but a new feature from upstream comes along which pushes the 4MB over the edge. What then? I would like to make no promises. Do the announcement and urge people to upgrade their hardware, as future releases may stop including builds for devices with 4MB flash at any time, depending on whether or not new firmware fits the old flash layout. No effort will be made to reduce the binary size (which becomes more and more time-consuming to achieve, if at all possible). I don't see an item in the web UI that tells me how much flash my ESP32 has. This should be available somewhere. Otherwise, some (few) people will buy new hardware even though they might not need to and some (many) will ask how they know whether or not they need to upgrade. Ripping the ESP32 board out of their setup to connect the ESP32 using serial and use PSRAMLet's keep this optional. I guess it makes sense to explain people that any and more PSRAM is better, and that it makes little difference in the price, but might make a big difference regarding how many features they can use at once. Example: the prometheus web API endpoint might need so much memory, depending on the amount of features enabled, that it becomes difficult to serve it reliably. I suppose that even without PSRAM, most users should be good to go. ESP32-S3If we start urging people to use a particular ESP32, they should choose an ESP32-S3 with the USB-port directly connected to the USB-subsystem of the ESP32-S3. That way, 3 hardware UARTs are available. Community Boards: OpenDTU FUSION and @swingstate's PCB (which other relevant boards are there?)I asked people from AllianceApps, who confirmed that all OpenDTU FUSION boards (at least starting from v2.x) are equipped with 8MB flash (and 2MB PSRAM, to be precise) at least. As far as I can tell, @swingstate's PCB uses an ESP32-S3 with 16MB flash. I guess it is early enough to make him use N16R2 (2MB PSRAM) modules (or N8R2). Proposed Next Steps
Can I get some thumbs up or criticism? @helgeerbe What are your thoughts on this? Are you prepared to upgrade your hardware or would it annoy you 😉 (or are you on a 8MB ESP32?) |
|
just to get a confirmation, is this a viable option? WROOM-1-N16R8 ESP32-S3-DevKitC-1 module |
|
Agree to the steps @schlimmchen shared above. My PCB is based on the ESP32-S3-WROOM-1-N16R8, 16MB Flash / 8MB PSRRAM. I should also hear from TÜV next week on next steps. Would like to progress faster with the board evaluation but doing a CE cert right is a pain for a single man show. Regarding point #2, I'd suggest to announce the drop of 4MB for a certain quarter in the future, for example Q4. That should provide enough time for users to get a new ESP. Entering the quarter we should have visibility on which release would be the "final" for the 4MB boards. |
should work, however looks like the PINs are not soldered. Suggest you get a board with the PINs already soldered unless you are keen to do that and have the equipment. |
I can't read that, it Russian, dude 😁 From what I see, it fits. Please try to find out whether or not there is an additional UART<->USB transceiver on there, or if the USB lines of the ESP32 are directly connected to the USB plug. The latter is preferred such that the third hardware UART is available for use. |
|
oh, i’m using the english version of the ali, have no idea why it copied me the russian one. apologies! will try finding that but the description is usually pretty poor on ali listings. |
So changing the flash layout to not include a second full-size OTA partition is no longer an option?
Would this open up the possibility to use a CMT2300 module and the Huawei CAN interface on a single board? Because that would at least cut my need for new ESPs in half, if 4MB support gets dropped for real. |
Exploration is required to determine the feasibility. Currently I think that it will not be possible for a simple reason: As far as I can tell the ESP32 executes instructions from flash (on a full-sized computer, instructions are copied to RAM and executed from there. since this cannot be the case on an ESP32, my best guess is that ESP32 executes code in-memory (flash)). In turn this means: Without complicated locking mechanisms or without complicated "copy some instructions to RAM and execute from there", we are simply not able to update the currently running firmware in-situ. If you show us how this actually could work without significant implementation effort, we can reconsider this. Until then, I regret putting this option on the table, for the fact that I think to have underestimated the overhead. Also: I underestimated the willingness of the most active users to throw support for 4MB flash devices out the window. Even more so: I know that a new project exists whose author aims to only support 8MB+ flash devices from the very beginning. The overall trend seems to be that 4MB ESP32 are simply outdated.
Nope. That's an SPI issue, not UART. |
|
Wink mit dem Zaunpfahl 😇
|
|
OK, I added a discreet comment about the future need of ESP32s with larger memory: https://github.com/helgeerbe/OpenDTU-OnBattery/wiki/ESP32-Versions-and-Memory |
|
I just ordered and will update all my boards with this one --> https://www.lilygo.cc/en-pl/products/t-eth-lite (cheaper than ali). So hell yeah, up the compatibility matrix and refrain from the nonsense work of making it fit. |
|
That thing looks nice, however, how is that receiving software? Does one need to connect an external TTL-UART<->USB adapter to flash the initial software? |
Yep, exactly (1Euro ish @ ali). Its a one-time procedure - same thing like the WT32-ETH01 init (which currently services opendtuob here). Just connect 5 cables, flash, done. And BTW - thanks a million for the amazing webflasher that I'll give a first shot then =) |
|
is the web flasher limited to german ips? it’s unavailable for me, for whatever reason. |
|
@greymda |
|
@MetaChuh as per the usual, it was not working for days and now it is 😀 |
|
perfect. greetings, |
|
Hallo, ich habe ein paar von den Dingern. Die Variante im ersten Bild mit dieser Antennenform auf dem im unteren Bereich P2N8
Die Version hier , mit der selben Antenne und im unteren Bereich aufgedruckten MON16R8 scheint zu funktionieren.
Die folgende Version mit einem anderen Antennensystem funktioniert bisher am besten.
Ich denke das die fehlerhaften Versionen inzwischen ohnehin nicht mehr verkauft werden. Die hatte ich ende 2023 gekauft. |
|
@Solar320 in unserer solar community versuchen wir gerade alle user auf das einheitliche community fusion board design (von der community und für die community) zu leiten, damit wir alle weniger support anfragen, sowie neue user weniger frusterlebnisse haben. thx & greetings, |
|
Ok, da muss ich mal suchen ob ich was dazu finde. Auf die schnelle finde ich hierzu nix bei Github. Ich vermute das das eine art Mainboard ist auf dem die ESP-S3 draufpassen und diverse ADUM chips für Canschnittstellen und außerdem die jeweiligen Funkmodule usw..... |
|
Du hast nichts gefunden?! 😲 https://shop.allianceapps.io/products/opendtu-fusion-community-edition Mit diesem Board wirst du jedenfalls wenig Frust haben, weil die einfach funktionieren, insbesondere der RF Teil. |
|
Naja nicht direkt bei Github. Über google jetzt schon ;) Das Board sieht sehr gut aus und ist auch relativ günstig, allerdings habe ich schon NRF24 und CMT Module gekauft. Meine DTU existiert derzeit noch als Drahthaufen auf dem Tisch, da ich es noch nicht geschafft habe ein 3D Druck Gehäuse dafür
Das ESP32-S3-WROOM-1U scheint sogar N16R8 zu haben, wenn es davon nicht noch weitere Varianten gibt. Mal sehen, vieleicht kaufe ich so ein Board um mir den ganzen Verdrahtungsaufwand und vor allen Platz im Gehäuse zu sparen. |
Genau deshalb gibt es die FUSION Platine. Breadboards bzw. diese Kabelbrücken sind super zum Entwickeln bzw. Evaluieren, aber totaler Mist auf Dauer, weil das einfach nicht zuverlässig ist.
Dafür gibt es #955. Oder auch jetzt schon im Shop mit nur einem ADUM1201. |
|
Ich hab bei mir letztens auch auf eine S3 N16R8 umgerüstet, um für die ODoB Zukunft gerüstet zu sein. 😄
Die Fusion Platine finde ich auch ziemlich geil, würde aber persönlich den Bastelspass vermissen. |
Ja das habe ich mitbestellt, leider ist das noch die alte Version, die neue hätte CAN und einen extra seriellen wohl alle isoliert. Hauptsache die Serielle ist darauf isoliert. "The v1 Prototypes which just have 1x ISO will be available in limited numbers as well much sooner, so if you have just a single Victron these might suit you well." Ok sollte isoliert sein. Für die anderen Schnittstellen könnte ich eigene Isolatoren verwenden wenn ich die mal brauchen sollte. @DonJohnLong Das Board selber ist auch derzeit nur mit N8R2 ausgerüstet obwohl es in 50er stückzahlen den ESP32-S3-WROOM-1U-N16R8 schon für ca 4,10/Stk gibt. Der Link funktioniert vieleicht nur in Österreich: |
|
Du hast eine Sache missverstanden: Die jetzt schon angebotenen CAN/ISO shields haben in der Tat nur einen ADUM1201 für VE.Direct, aber sie haben sehr wohl schon einen CAN Transceiver drauf. |
|
Hab ich jetzt auch gesehen. Bei der V2 version sollte aber dann jede schnittstelle auch einen eigenen Isolator haben, gerade bei so sachen wie Batteriewechselrichtern wo durchaus hohe potentialunterschiede auftreten könnten bzw. bei Potentialunterschieden zwischen verschiedenen angeschlossenen Geräten ist das ganz praktisch und man erspart sich viel ärger. Na mal sehen wann die Platine ankommt, dann muss ich mich halt mal aufraffen und noch ein Gehäuse drucken. |
@schlimmchen |
|
Du solltest dir das Basteln nicht verbieten lassen oder madig reden lassen, wenn du Spaß dran hast. Aber lass mich nochmal auf das OpenDTU Fusion Board hinweisen, was dir viel Zeit ersparen kann -- wenn du das möchtest. |
|
Sorry, hatte ich mißverständlich formuliert. Die Permanent PCBs sind für meine anderen Basteleien (Multikopter, Lichtsteuerung etc. pp.) vorgesehen. |
|
Thanks to everyone for their respective contribution to this discussion, helping to reach a decision which seems to be widely supported. The announcement is in #1025. |
|
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new discussion or issue for related concerns. |
After we finally hit the flash memory limit in #923 and had to thank Thomas Basler for serving us a solution to save some code duplication and hence flash memory on a silver platter, we need to have a discussion about how we handle the growing code.
One (aggressive) idea is to drop support for devices with 4MB flash altogether. I had this idea myself as well.
swingstate's comment:
I'm exploring the possibility of discontinuing support for devices with 4MB flash. Do you think increasing the minimum to 8MB would pave the way for new features or advantages in the near future? It might be worth communicating this potential change widely, such as in changelogs, the wiki, GitHub page, and other prominent spaces. For example: "Starting January 1st, 2025, ODTUoB will no longer support 4MB flash. Further details can be found here."
Personally, I believe phasing out support for these smaller devices could catalyze further development. What are your thoughts?
Best regards,
Originally posted by @swingstate in #923 (comment)
The text was updated successfully, but these errors were encountered: