Memory usage and optimization

[KlausMu]

Memory usage or lack of memory has been topic of several discussion in the past.
I would like to share the information I gained during my analysis and how the current situtation is, so that we can discuss how to proceed.

Different types of memory
First thing that needs to be known is that the ESP32 has different kinds of memory:

• SRAM 512 kB (SRAM means "shared RAM" and is used for different kind of RAM)
• Flash 4096 kB (similar to an SSD in a desktop computer)
• wrover models have additional 4096 kb of (slow) RAM called PSRAM

A simplified explanation is

• the program code goes into the flash (called IROM)
• Variables and static data goes into a part of the SRAM called DRAM

An interesting detail is, that const data will also automatically be placed by the linker into the flash and can directly be used at runtime from flash. This is called DROM (Data stored in flash). So if you have problems with too little RAM, you could move data to flash if it is constant.

This is also explained (in more detail) on this page about Memory Types from Espressif.

So now we know that there exists IROM, DROM (both in flash) and DRAM (in SRAM).

Flash: Size of the code
As already said, the code goes into the flash. The size of the code was a problem in the past, but is no longer, because we are using the "huge_app.csv" partition layout. If you are interested in that topic, I could write a litte bit more about this and why e.g. it is not possible to use OTA (over the air updates) with that partition layout.

DRAM
The interesting part is the DRAM. The best page I found explaining the DRAM is this blog about the "ESP32 Programmers’ Memory Model".

You don't need to read and understand it completely. Here is what is important for us (again simplified):

DRAM has several regions which share the same total available amount of DRAM (320 kb)

• static variables with fixed size (called static DRAM)
• heap: all variables allocated at runtime are here
• stack: if you call functions, the parameters are placed here

Total size of DRAM is 320 kB, and this has to be shared between the three above. So if you have more static variables, you will have less heap and vice versa. With one important restriction: static variables (static DRAM) can never have more than 124580 bytes.

code example	where it goes
const appleTvIcon_map[] = { 0x00, 0x00, ...}; (a lot of data)	IROM (part of the flash)
static lv_color_t bufA[ screenWidth * screenHeight / 10 ];	static DRAM (part of the DRAM)
lv_color_t * bufA = (lv_color_t *) malloc(sizeof(lv_color_t) * screenWidth * screenHeight / 10);	heap (part of the DRAM)

Limited static DRAM
As already said, static DRAM is limited to 124580 bytes. If you have more static content, then the linker will complain about it and is not able to create the file "firmware.elf". You will get a message like

section `.dram0.bss' will not fit in region `dram0_0_seg'
DRAM segment data does not fit.
region `dram0_0_seg' overflowed by xxx bytes

You have to reduce the static content in your code until the linker can link the "firmware.elf".

If you ever have to examine the content of the static DRAM and why it is at the limit or even above the limit, I can recommend this page.

Once I had a problem that with WiFi and BLE activated, the static DRAM was not big enough. I analyzed the static DRAM, which looked like this:

So the solution to this issue was to move bufA and bufB from static DRAM to the heap. Static DRAM has a fixed limit, but heap has not. Heap simply takes the rest, and this is in fact more than the static DRAM. Of course no memory was gained with that trick, because as static DRAM usage decreased, heap usage increased. It simply overcame the limit of the static DRAM.

Heap usage
Heap usage can only been seen at runtime, e.g. like that

#include "Esp.h"
Serial.printf("heapSize: %lu, heapFree: %lu, heapMin: %lu, heapMax: %lu\r\n", ESP.getHeapSize(), ESP.getFreeHeap(), ESP.getMinFreeHeap(), ESP.getMaxAllocHeap());

If you are running out of memory at runtime, normally the ESP32 will crash or at least will no longer work as expected.

---

Next part will be a detailed analysis on how many static DRAM and heap is used by the different software components used by OMOTE. I will do a separate post for this.

[KlausMu]

Now that we know how static DRAM and heap are related, we need to know why we sometimes have problems with memory and if there is anything we can do about it.

If is also important to know, that you will only know at runtime if heap is big enough. Sometimes the ESP32 even needs to run for some time until you know that there is not enough heap. E.g. WiFi is known to need memory peaks at runtime in heap. That is the reason why WiFi sometimes breaks during usage or even the whole ESP32 reboots.

Ok, here is the result of my measurements.

• all measurements were done with "framework-arduinoespressif32 @ 3.20009.0 (2.0.9)"
• lvgl version 8.3
• BLE means use of NimBLE, not the standard bluedroid based library. With the standard BLE library, memory usage is much worse (65% higher) and, most important, it cannot be used at the same time as WiFi.

library / function used	Static DRAM (124580)	heapFree (349076)	Δ heapSize (374612)	Δ heapFree (349076)	heap peak	comment
WiFi (including MQTT)	21660	260252	41568	88824	9960	peaks up to 10 k (4.4 k more than usual)
BLE	13128	226168	84104	122908	196
BLE + WiFi	30920	153896	108184	195180	14228	peaks up to 14 k (8.6 k more than usual)
IR	8	349068	8	8	5588
IR incl. receiver	656	342500	1972	6576	6068	peaks up to 6.1 k (0.5 k more than usual)
IMU accel	440	345988	2184	3088	5584
keypad	296	348780	296	296	5588
TFT, incl. dimming via lcd channel and touch (FT6206)	960	343824	3776	5252	5588
lvgl	1252	346540	1448	1448	5588	lvgl itself needs not so much memory in static DRAM and heap, but preallocates a lot for internal memory management
lvgl: preallocated static work_mem_int	32768		32768	32768
lvgl: static bufA, bufB, 10%	30720		30720	30720

So the more static DRAM is used, the less is left for heap (heapSize) - as we expected.

The worst memory consumers are (in that order):

• BLE
• WiFi
• the statically or dynamically allocated memory needed for lvgl

WiFi and BLE take more from heapSize as it should be only from using some memory from DRAM. So they also allocate a lot at runtime in heap.

What does this now mean for real life? During development, we have to keep an eye on:

• static DRAM must not get bigger than 124580
• heapFree should stay big enough for runtime

There is nothing we can do about BLE and WiFi.

If only one of them is used, we have no problem with memory needed by lvgl. We can give lvgl more than the 32k it has by default, and this can be done both in static DRAM and heap. Both works, and I will explain later how to do it. With that you can have more than 100 tabs.

But as soon as both BLE and WiFi are used, things get complicated. I will write more about this in my third post. It will be about how to optimize lvgl memory usage.

[KlausMu]

Ok, this post will be about:

• how is lvgl doing memory management
• how can we tweak it

There are two different ways of memory management lvgl can use:

• lvgl's own heap implementation: a static memory, preallocated in static DRAM, with fixed size
• use of the system heap: allocated at runtime in system heap, no fixed size, only limited by available system heap

How can we tell lvgl what to use:

Way 1: lv_conf.h
Default setting is lvgl's own heap implementation, 32K in size:

#define LV_MEM_CUSTOM 0
...
    #define LV_MEM_SIZE (32U * 1024U)          /*[bytes]*/

If you need more, you can increase "32U". If you get section '.dram0.bss' will not fit in region 'dram0_0_seg', then it was too much (see first post).

If you want to use the system heap, it would look like this:

#define LV_MEM_CUSTOM 1
...
    #define LV_MEM_CUSTOM_INCLUDE <stdlib.h>   /*Header for the dynamic memory function*/
    #define LV_MEM_CUSTOM_ALLOC   malloc
    #define LV_MEM_CUSTOM_FREE    free
    #define LV_MEM_CUSTOM_REALLOC realloc

The last four lines are automatically activated.

If you want to use PSRAM (external RAM, only if you have an esp32-wrover), there is also an example in file "lv_conf.h". In that case lvgl also preallocates its own heap, but in the PSRAM, not in the static DRAM.

Way 2: platformio.ini
For me, this is the preferred way, because if I were doing changes in "lv_conf.h", sometimes they were not recognized and I had to do a manually "clean". If the settings were in "platformio.ini", changes always got recognized and a complete rebuild was automatically triggered. My settings look like this:

build_flags = 
    ; Variant 1:
    ; Don't use lv_conf.h. Tweak params via platfom.ini. See lv_conf_internal.h line 31
    -D LV_CONF_SKIP=1
    ; use millis() from "Arduino.h" to tell the elapsed time in milliseconds
    -D LV_TICK_CUSTOM=1
    ; dynamic memory. Takes as much as it gets from heap (DRAM)
    ;-D LV_MEM_CUSTOM=1
    ; static memory, will be allocated in static DRAM
    -D LV_MEM_CUSTOM=0
    -D LV_MEM_SIZE="(72U * 1024U)"
    ; other settings
    -D LV_FONT_MONTSERRAT_12=1
    -D LV_FONT_MONTSERRAT_16=1
    -D LV_FONT_MONTSERRAT_24=1
    -D LV_USE_CALENDAR=0
    -D LV_USE_IMGBTN=0
    -D LV_USE_WIN=0
    -D LV_THEME_DEFAULT_DARK=1
    -D LV_BUILD_EXAMPLES=0
    ; Enable the log module
    -D LV_USE_LOG=1
    -D LV_LOG_LEVEL=LV_LOG_LEVEL_ERROR
    ;-D LV_LOG_LEVEL=LV_LOG_LEVEL_TRACE
    -D LV_LOG_PRINTF=1
    ; ---
    ; Variant 2:
    ; or define where lv_conf.h is, relative to the `lvgl` folder
    ;-D LV_CONF_PATH=../../../../src/lv_conf.h
    ;------------------------------------------------------------

Logging
If lvgl runs out of memory, depending on the memory configuration, you get more or less information in the serial output. With default log settings and using the internal heap, you get no message at all. Nice. In that case lvgl simply stops to work, and you don't know why. That's why I highly recommend to activate logging as you can see in my "platformio.ini".

How to examine memory usage at runtime
This depends on which memory management lvgl was told to use. You can use this code snippet, which works in both cases (lvgl's managed static heap vs. system heap)

#include "Esp.h"
#include <lvgl.h>

#if LV_MEM_CUSTOM == 0
lv_mem_monitor_t mon;
void log_memory(){
  lv_mem_monitor(&mon);
  Serial.printf("total_size: %lu, used: %lu (%3d %%), free: %lu, max_used: %lu, free_biggest: %lu, frag: %3d %%\r\n", mon.total_size, mon.total_size - mon.free_size, mon.used_pct, mon.free_size, mon.max_used, mon.free_biggest_size, mon.frag_pct);
}
#else
void log_memory(){
  Serial.printf("heapSize: %lu, heapFree: %lu, heapMin: %lu, heapMax: %lu\r\n", ESP.getHeapSize(), ESP.getFreeHeap(), ESP.getMinFreeHeap(), ESP.getMaxAllocHeap());
}
#endif

Now we are ready to go and to see what is going on behind the scenes of lvgl ...

[KlausMu]

The fourth post will be about

• how much memory is used by lvgl
• how to optimize memory usage of lvgl

Test 1: 100 tabs, static DRAM vs heap
The first test was creating 100 tabs at runtime, each having one label and one slider in it.

static DRAM	heap
68568 bytes	78720 bytes

Wow, heap needs 25% more memory than lvgl's own heap implementation. I didn't expect this.
I didn't find a lot documentation about lvgl's own heap implementation, only this post.

Both are safe. One uses LVGL’s own heap implementation; the other relies on you providing heap functions that LVGL can use.
If you already have a system heap (e.g. on Linux) there is no need to use LVGL’s implementation.
Also, on embedded systems the LVGL allocator is sometimes better at avoiding fragmentation than what the platform provides.

Test 2: how much memory does a single slider or a single label need
A single slider (without any styling) needs about:

static DRAM	heap
302	359

A single label (without any styling) needs about:

static DRAM	heap
136	171

Again, dynamic memory (heap) needs 27%-30% more memory.

So is the heap good for anything?
Yes, if you want to have so many lvgl objects that they would never fit into the limited static DRAM.

100 tabs with 11 labels in each tab needed 213656 bytes in heap. This never would have been possible in static DRAM (124580 bytes max)

In my further tests, when I only provide a single byte number for memory usage, this is always for static DRAM (lvgl's own heap implementation)

Test 3: swiping through tabs
To make it short: I swiped both programmatically and manually through a lot of tabs, and although memory usage goes up and down by some bytes, overall swiping does not permanently allocates memory.

Test 4: styling of objects
Styling a label with a single line like lv_obj_align(myLabel, LV_ALIGN_TOP_LEFT, 0, 20); needs 56 bytes

Test 5: memory usage of "real live tabs"
The four tabs available in the demo of @CoretechR need this amount of memory:

object	bytes
tabview	668
4 tabs (initially empty)	612
settings	4620
numpad	4720
appleTV	936
smartHome	2860
pageIndicator	2136 (for three tabs)

Interim conclusion
If you don't use WiFi and no BLE, you can maybe allocate 20 tabs having the size of the numpad tab in static DRAM and maybe 50 tabs in heap.
If you use WiFi and BLE, well, then the limit is more likely 4 or 5 (your own code also takes some bytes). This is bad news for the discussion about scenes, where we plan to have a lot of tabs.

Ok, what could we do:

• use PSRAM: I heared that it is slow. And I don't have a wrover. And the current OMOTE hardware had to be changed.
• at least, don't use BLE. We could have an "OMOTE satellite" which connects as a BLE keyboard to the media player. The OMOTE talks to the satellite via MQTT. The satellite could also send the IR signals. I think this worked for sure, but would make things complicated
• dynamically create and delete the lvgl tabs. Why should we have 100 tabs in memory, if only one (and the one to the left and the one to the right during swiping) can be seen?

Let's give the third option a try.

Test 6: dynamically create content (screen, tabview and tab)
I see three ways of dynamically creating the content:

a) create and delete the tabs
b) create and delete the tabview
c) create and delete the screen

I tested it with only creating these three objects, with no content in it (no labels etc.)
a) didn't work well. Memory got not freed completely, and the tab didn't disappear from the tabview, even when it was deleted.
b) and c) both worked well. Memory usage stayed at a constant level. That is what we want to have.

So I decided to go on with option b) delete the tabview.

Test 7: dynamically create content
Let's see if this works well if the tabs have real content in it.
So I took again the four tabs (settings, numpad, appleTV, smarthome) and placed them on only three tabs, because only three of them can be seen at the same time.
As soon as a swipe finished, the whole tabview was deleted and recreated with three tabs being visible after the swipe.
Also the swiping was done programmatically, so I could let this run for a long time. Memory stayed constant both with static DRAM and heap.

It was not noticable that a tab was deleted and recreated (at another position in the tabview) after swiping. No flicker, nothing.

So this looks promising. With that trick, you can have hundreds of tabs (in the code), and only 3 of them are in memory at the same time. And there should be enough memory for 3 tabs ...

I read about this trick in the lvgl forum and that it is the best way to keep memory usage low, but without any advice on how to do it.

Downsides
I saw two downsides.

As the tabs are deleted and recreated at runtime, the state will not be preserved. So if you have a slider or a switch, the state gets lost. For this you have to give the tab the possiblity to save it's state before it gets destroyed. When the tab is recreated again, the tab can restore its previous state. In my tests this worked well. Only "settings" and "smarthome" are using widgets having a state (the sliders and switches).

The other downside I saw is that some widgets cannot be created with an inital state. A slider for example can be created with an initial value. But a switch can only be created "off", and you have to set it to "on" afterwards. And this indeed can be seen as a flickering of the switch.

Still open
When using manually sliding (not programmatically as in my tests), we need an event when the sliding ended. I still don't know how to do it. It was possible in lvgl 7, but I didn't get it running in lvgl 8. I asked in the forum, but no answer until now. Maybe someone here can jump in.

Conclusion
To me the dynamic creation of tabs looks promising. At least in my prototypes it worked very well.
Yes, this all adds complexity, but with the benefit of being able having hundreds of tabs.

What do you think about it? Any ideas or comments about this?

[MatthewColvin]

First thing first, That was an awesome breakdown of memory!! I have been wanting to research this for a while now and I really appreciate you taking the time to do such a nice write up.

A few random thoughts to some of the topics in your posts.

I like how you called out replacing the lv_config.h for the platform.ini file. I think in my branch I actually completely eliminated the need for the lv_config.h at all. https://github.com/CoretechR/OMOTE/blob/abstraction/Platformio/platformio.ini

I was also toying with this idea of dynamic creation to allow for more UI and that is part of the reason I did this wrapping of LV objects in
https://github.com/CoretechR/OMOTE/tree/abstraction/Platformio/OmoteUI/core
Although Looking at it now I am not sure how much I like how it actually looks when constructed.

You mention one issue of dynamic creation being state and we have had some conversations in the past about how we should be storing data in non-volatile memory but nothing has really ever come of those conversations.

We really need some sort of light weight module that can use non-volatile memory to store all the info needed for the UI. I have looked at a few solutions to this and there are some modules that implement a simulation of EEPROM using the Flash. There are also some examples of carving off a portion of flash to create a filesystem. https://randomnerdtutorials.com/esp32-littlefs-arduino-ide/

I was thinking maybe we could store a JSON config in this littlefs that could be dynamically updated to save the current state of many things so that way the UI could be inflated off it.... But this is no small undertaking lol.

[KlausMu]

I like how you called out replacing the lv_config.h for the platform.ini file. I think in my branch I actually completely eliminated the need for the lv_config.h at all. https://github.com/CoretechR/OMOTE/blob/abstraction/Platformio/platformio.ini

Guess where I have this idea from :-)

You only did some minor mistakes:

-D LV_CONF_INCLUDE_SIMPLE

This tells lvgl where lv_conf.h can be found - at a "simple place", which means some default paths. But we don't want to use it at all.

-D LV_MEM_CUSTOM=1
-D LV_MEM_SIZE="(48U * 1024U)"

This combination does not make sense. With the first you activate usage of system heap, with the second you set the size of the internal, lvgl managed heap. So the second argument is ignored.

I was thinking maybe we could store a JSON config in this littlefs that could be dynamically updated to save the current state of many things so that way the UI could be inflated off it.... But this is no small undertaking lol.

Yeah, I thought about this as well.
In the "huge_app.csv" partition we currently are using, both nvs and spiffs are still available:

# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x5000,
otadata,  data, ota,     0xe000,  0x2000,
app0,     app,  ota_0,   0x10000, 0x300000,
spiffs,   data, spiffs,  0x310000,0xE0000,
coredump, data, coredump,0x3F0000,0x10000,

nvs is the "non volatile storage", used for key/value pairs. We already use it for the preferences. And WiFi is also using it - and you never know how much of it. I used it in the past for another project and had to increase this partition. It worked well, but it is not very fast.

The spiffs partition currently is not used at all by OMOTE. 917504 (= 0xE0000) bytes of the flash, almost 25%, currently used for nothing :-)

I think the software part for using spiffs is not a big thing. But another library would be needed which takes memory ... But surely worth having a closer look.

[bittnert]

Dear @KlausMu
thanks for the detailed analysis. You confirmed my suspicion that we have memory issues (at least sometimes).

I agree with the approach you proposed. However, what I don't fully understand is, why we want to have all scenes active on tabs at all times. I currently have a harmony remote and my parents are using two. On the harmoy remotes, you can always only have one scene (they call it activity but I like scene better) active at any given time. When you start a new scene, it would close the currently active one.

However, what the harmony remote can do (and I'm not sure we can do the same easily) is, that devices which are active in one scene and are needed in the next scene are kept active instead of being turned off and on again.

BR
Thomas

[KlausMu]

I agree with the approach you proposed. However, what I don't fully understand is, why we want to have all scenes active on tabs at all times.

No, of course only one scene would be active. What I meant is that you don't want to have all tabs in memory, even those from the scenes not being active. So we need the dynamic creation of tabs because of two reasons:

• memory
• tabs have to be reordered when the scene changes, because the tabs are always in a linear order, but depending on the scene most likely at different positions

However, what the harmony remote can do (and I'm not sure we can do the same easily) is, that devices which are active in one scene and are needed in the next scene are kept active instead of being turned off and on again.

That's easy to achieve. You will know which scene you are coming from and where you are going to. As switching the scene is currently done programmatically, you can do whatever you want in code. As soon as everything might be configured via files, things get more complicated. Here you would need to know explicitely which devices are involved in the scenes, so that you can decide which devices can stay on and which have to be powered off.

[KlausMu]

Update: in lvgl 9, it is no longer

LV_MEM_CUSTOM=0
or
LV_MEM_CUSTOM=1

but

LV_USE_STDLIB_MALLOC=0
or
LV_USE_STDLIB_MALLOC=1

And also some other build flags taken from lv_conf.h have changed. See the updated platformio.ini of the OMOTE project. (as of writing this, not yet published, but you will see when OMOTE switched to LVGL 9)