-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory leak in sink #6
Comments
Played around a bit today and seems like the root cause is related to the V8 implementation (or even the spec according to the literals) of I confirmed that when substituting |
Thanks for the diagnosis @betamos - any ideas for how to fix? |
I played around a bit with the code but didn't quite get it to work, so I ended up hot-swapping I know that there are more features in this module than in I think the key is to create an internal source which yields both the messages from the original source and control messages (like error and finish), generated from plain old callbacks. It seems like one has to avoid |
Details for why the leak happens are in the issue, #6. This code is heavily influenced by the fix for writeToStream in streaming-iterables but avoids breaking any functionality here. - Fix leak by removing Promise.race use - Adds the [leak reproduction code](#6 (comment)) to a benchmarks folder. Run instructions leveraging Clinic.js are inline in the file. I can add to the readme if desired. - Updates deps :) - Added `package.files` to restrict what gets uploaded (tests were being packaged) - Travis now runs Node.js `stable`(14) and `lts`(12) ## Clinic.js Reports I've also listed the IPFS CIDs (linked to the gateway) if you want to peruse in detail. **After** [bafybeidzy5otcicczh43ifdcg55btdl4q5j75vfolhsygr3oomqjavj6p4](https://gateway.ipfs.io/ipfs/bafybeidzy5otcicczh43ifdcg55btdl4q5j75vfolhsygr3oomqjavj6p4) ![image](https://user-images.githubusercontent.com/639834/87229565-4ef83480-c3a9-11ea-8b62-222c83931fa6.png) **Before** [bafybeig3sw5t326is6e7rknf7ena3ffubekperpsdgf2dty7w74pkvmhai](https://gateway.ipfs.io/ipfs/bafybeig3sw5t326is6e7rknf7ena3ffubekperpsdgf2dty7w74pkvmhai) ![image](https://user-images.githubusercontent.com/639834/87229558-37b94700-c3a9-11ea-8b4a-9d5ca2318f14.png) ## Package file changes ```sh $ npm pack --dry-run npm notice npm notice 📦 [email protected] npm notice === Tarball Contents === npm notice 1.1kB LICENSE npm notice 151B duplex.js npm notice 215B index.js npm notice 2.2kB sink.js npm notice 494B source.js npm notice 395B transform.js npm notice 1.1kB package.json npm notice 3.4kB README.md ``` fixes #6
It looks like
sink
has a memory leak. Repro:Memory consumption steadily increases until completion and stays at around 800-900 Mb. Total size of those buffers are around 700 Mb. Early diagnostics with chrome webtools point towards the promises and/or the callback hijacking of
stream.end()
.Found when using the upstream
libp2p-js
withlibp2p-js-tcp
.node v13.11.0
Linux 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
The text was updated successfully, but these errors were encountered: