Buffer for DynamicJasonDocument livedata/status and prometheus/metrics too big? #739
Comments
|
have you tested it with 10 inverters and all descriptions filled to it's max size? |
No I don't. Do you have an example json, which uses the maximum size in settings? Or can you tell me, what the maximum size for liveview and prometheus output is? So I could copy it into one test OpenDTU |
|
You can copy the output of one inverter into the assistant and look at the memory consumption: https://arduinojson.org/v6/assistant/#/step1 Every description field can be 31 characters long. that means 10 inverters * 31 chars + 4 channels / inverter * 31 characters = is already 1550 b only for texts... then you have the values itself and the overhead for the json output itself. |
|
I see. I tested it on your latest originalOpenDTU release from master. With 10 HM-600, so only 2 channels. Calculator says 24576bytes. Starts up fine Without using prometheus I see in live view: Sometimes I see a crash in the network loop: I added: MessageOutput.printf("MaxAllocHeap: \"%d\".\r\n", ESP.getMaxAllocHeap());
DynamicJsonDocument root(40960);Documentation says that you can use I checked the code you alloc theDynamicJsonDocument, serialize Jason, and do all the network things. It might be better to free the DynomicJasonDocument directly after serialization. try {
String buffer;
{
MessageOutput.printf("MaxAllocHeap: \"%d\".\r\n", ESP.getMaxAllocHeap());
DynamicJsonDocument root(40960);
JsonVariant var = root;
generateJsonResponse(var);
serializeJson(root, buffer);
}
MessageOutput.printf("MaxAllocHeap after freeing JsonDocument: \"%d\".\r\n", ESP.getMaxAllocHeap());
if (buffer) {
if (Configuration.get().Security_AllowReadonly) {
_ws.setAuthentication("", "");
} else {
_ws.setAuthentication(AUTH_USERNAME, Configuration.get().Security_Password);
}
_ws.textAll(buffer);
}
} catch (std::bad_alloc& bad_alloc) {
MessageOutput.printf("Call to /api/livedata/status temporarely out of resources. Reason: \"%s\".\r\n", bad_alloc.what());
}
_lastWsPublish = millis();
}
}And I don't see any bad_alloc notification in the last couple of minutes. But I still received the above mentioned crash a couple of times. Tracking down to this code. Exception for new is not handled correctly. int WiFiUDP::parsePacket(){
if(rx_buffer)
return 0;
struct sockaddr_in si_other;
int slen = sizeof(si_other) , len;
char * buf = new char[1460];
if(!buf){
return 0;
}For testing I changed it to: nt WiFiUDP::parsePacket(){
if(rx_buffer)
return 0;
struct sockaddr_in si_other;
int slen = sizeof(si_other) , len;
char * buf = 0;
try{
buf = new char[1460];
} catch (std::bad_alloc& bad_alloc) {
return 0;
}
if(!buf){
return 0;
}Doing this hack, I don't see any memory issues any more. |
|
I am currently doing some tests here as well.... I also added 10 inverters. I am calling the prometheus endpoint every 5 seconds. The LiveView is running in two browser windows. The Memory Info shows the following info: With only 2 inverters the memory usage is at about 125kb. How long are you running this setup until you receive such errors? |
|
With the latest merge of 23.5.22 into OpenDTU-onBattery no liveview was possible anymore due to memory fragmentation. As a workaround I set |
|
Would close this issue as it is related to very old code now. the structure of the json output was also changed |
|
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new discussion or issue for related concerns. |
What happened?
I use liveview and prometheus very frequently and sometimes liveview can't update and the page goes blank due to bad_alloc error. If things goes worse, the system never recovers and I have to reboot to see livedata again.
I run my own OpenDTU with some extensions, so this might be an edge case? Don't know.
But I observe, that for both api calls from my point of view a far too big buffer is requested.
DynamicJsonDocument root(40960);I don't know the max possible size for a response, but for my setup 4096 bytes for the livedata will be enough. So OpenDTU requested a 10 times bigger buffer.
I reduced the buffer to 20480, and since then I can't observer any bad_alloc error anymore.
To Reproduce Bug
Run liveview and prometheus in parallel. Wait on console for the out of resources output.
Expected Behavior
No bad_alloc errors due to memory fragmentation.
Install Method
Self-Compiled
What git-hash/version of OpenDTU?
3f8226c
Relevant log/trace output
No response
Anything else?
No response
The text was updated successfully, but these errors were encountered: