CPU load grows in time

[maskac]

NS 14.0.4 , docker hosting for 150 users
after upgrade from 13.0.1 to 14.0.3 than 14.0.4 CPU load grows in time

With 13.0.1 no problem occured.

[sulkaharo]

Found a memory leak, fixing. Thank you for the report - the increase on my single instance monitor is so small I didn't notice this...

[sulkaharo]

Based on monitoring the data loads reported from #6133, looks like this fixes the leak. Can you help validate the fix asap, so I can push this out as a point release?

[maskac]

Now upgrading all nightscouts, will be report in few hours,

[sulkaharo]

Awesome, thank you! ❤️

[sulkaharo]

Any news yet? Looking at my instance, I don't see the memory use going up anymore.

[maskac]

I don't know yet. It's to early for decision.

[maskac]

[sulkaharo]

What's the memory use before you updated, or is that shown on the graph?

[maskac]

Memory was small problem, CPU load was big problém. At 21:00 i restarted all nightscout. At 13:00 upgrade was made.

[maskac]

first screen shot bad. this is good.

[maskac]

Small peaks are backup process. Grow of cpu load is from nightscout.

[sulkaharo]

Right ok, didn't at first notice the creep is that slow. Would it be possible for you to add memory monitoring that tracks the node processes? The CPU use in NS is 100% dependent on the data in the runtime and if CPU is creeping but memory use isn't, that implies the memory use is for the VM instances which allocate more than Node needs and the true Node memory use is not actually tracked.

[maskac]

I not have agent inside, memory use is about 80-100MB / container.

[sulkaharo]

Right - given I got a couple confirmations that this seems to at least fix the issue visible in the logs, I merge this to dev and released as 14.0.5, as this should also fix the data backfills. Let's keep this open and please report back here when you have longer term results.

[sulkaharo]

How does the data look now? Also ping: I released 14.0.6 to fix an issue with batch upload of data from Loop, so you might want to update again. This resolves the outstanding big issues so hopefully there won't be more updates in the next few weeks.

[maskac]

I looking before few minutes ago and i don't know

Upgrade to 14.0.5 was made about 20:00. Peak si backup process. From 8:00 may be growth of load....

[sulkaharo]

If you compare that graph to the one on the top where there was clear 10% increase in the average over 24 hours, it looks to me like at least the bad resource leak is gone

[maskac]

CPU load & memory load sounds good with 14.0.5

Good job, Thanks.

[sulkaharo]

Great! Btw .5 has a small bug that break Device status uploads for Loop users, fixed in .6, so you might want to update once more. Apologies for that.