Queued buffers seemingly not cleaned up in closed sockets

[Euregan]

• Node.js Version: 12.9.1
• OS: Debian 10

We have a websocket server that seems to be leaking RSS, quite like in nodejs/node#4779. I also tried the fixes in #1518, to no avail. For around 500 sockets at the same time (Very short lived, a few seconds at most usually), we can go to up to 5G in RSS, while the heap used stays at around 800M. My observations are pretty much the same as in nodejs/node#4779.

Observations

• Slow clients (clients receiving data slower than server sends it) causes socket to increase their bufferSize
• This typically increases the RSS of the node process
• Upon disconnections of clients, we expect RSS to fall
• The RSS grows to up to ~7G across 3 instances (The server has 8G), and then stays there until it gets killed for using too much when it goes a bit above.
• The RSS is order of magnitude bigger than the actual Socket.bufferSize

Reproduction Steps

Test cases are in https://github.com/euregan/node-ws-slow-client-buffer-leak repository
Run server on a separate machine over the network
Run 200 clients for 30 seconds
RSS builds up.
Disconnects clients.
RSS stays built up

Graphs

Below are some graphs of the 3 instances. Sharp drops happen when the instance gets restarted either by pm2 or the system

Memory

Red is RSS
Yellow is Heap Total
Blue is Heap Used

Buffer size (bufferSize)

Websockets connected

[bnoordhuis]

Did you check with the inspector or node-heapdump that there's no leak in your program?

If heapdumps don't show anything but RSS peaks and won't come down, you might be running into (a variation on) nodejs/node#8871 or nodejs/node#33790 - tl;dr glibc memory fragmentation.

Switching to jemalloc can help, see the second issue.

[Euregan]

Thanks for the quick response, heapdump were not showing anything indeed, since the heap total was never too high.
We switched to jemalloc yesterday, that seems to do the trick, though we can't figure out how to make it work with pm2. I'm closing the issue, but maybe a warning could be added to the doc? Should I make a PR?

[eladnava]

@Euregan by running the following command, you can get pm2 and node to use jemalloc for memory allocation:

sudo -i
echo "/usr/lib/x86_64-linux-gnu/libjemalloc.so" >> /etc/ld.so.preload