test: log before and after RSS in memory leak test #21080
Conversation
|
Suggesting to fast-track this. |
|
+1 to fast-track |
|
@Trott can I take this as you approving fast tracking? |
|
Re-run Linux CI because debugging another issue left around some root-owned files causing a build to fail: https://ci.nodejs.org/job/node-test-commit-linux/19255/ |
Yes. |
|
https://ci.nodejs.org/job/node-stress-single-test/1893/ once it starts (predicted: 2 hours from now.) |
|
Linux re-run is green. AIX might take a while to start because I'm currently using one (and possibly the only right now) AIX hosts for a stress test. But it will start! |
|
(By the way, I think @tniessen already got results from this in a CI run off a branch. So the delay in landing this hasn't affected it's usefulness.) |
|
This is the output from https://ci.nodejs.org/job/node-test-commit-linux-containered/4862/nodes=ubuntu1604_sharedlibs_debug_x64/console, its like 5.097MB, so just above the limit. My suggestion in the referenced issue should solve the problem, I can open a PR tomorrow. (Even though I am not sure whether the problem reproduces with the recent v8 update.) |
It transpires that the extra bookkeeping in debug builds sometimes makes the increase in RSS go _just_ over the 5 MB limit, by fewer than 100 kB. Double the limit so we hopefully don't run into it any time again soon. The memory leak it tests for was one where RSS grew by hundreds of megabytes over the lifetime of the test; 5 vs. 10 MB is insignificant. Fixes: nodejs#21076
|
I added another commit that doubles the slop limit, PTAL. https://ci.nodejs.org/job/node-test-pull-request/15221/ |
There was a problem hiding this comment.
If I remember correctly, you originally wrote the regression test, so I'll trust your judgement on this one, but in my patch I also increased the iteration count: assuming there is a memory leak, it might go undetected in release builds if the amount of memory leaked per iteration is sufficiently small (which might not be the case, I didn't go into depth in the original issue). The patch which I showed to @Trott yesterday used 5e5 iterations and a limit of 8MB, which i assumed were reasonable choices.
|
Forgetting to free a DH key's memory leaks kilobytes at a time so 50k iterations should be plenty to flush that out. I picked 50k because otherwise it gets really slow with FIPS builds, because you can't use keys < 1024 bits. |
|
Okay, thanks for the explanation, I wasn't sure which part was being leaked. In that case this sounds reasonable. |
Refs: nodejs#21076 PR-URL: nodejs#21080 Reviewed-By: Tobias Nießen <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Anatoli Papirovski <[email protected]> Reviewed-By: Сковорода Никита Андреевич <[email protected]> Reviewed-By: Yuta Hiroto <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Tiancheng "Timothy" Gu <[email protected]> Reviewed-By: Luigi Pinca <[email protected]>
It transpires that the extra bookkeeping in debug builds sometimes makes the increase in RSS go _just_ over the 5 MB limit, by fewer than 100 kB. Double the limit so we hopefully don't run into it any time again soon. The memory leak it tests for was one where RSS grew by hundreds of megabytes over the lifetime of the test; 5 vs. 10 MB is insignificant. Fixes: nodejs#21076 PR-URL: nodejs#21080 Refs: nodejs#21076 Reviewed-By: Tobias Nießen <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Anatoli Papirovski <[email protected]> Reviewed-By: Сковорода Никита Андреевич <[email protected]> Reviewed-By: Yuta Hiroto <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Tiancheng "Timothy" Gu <[email protected]> Reviewed-By: Luigi Pinca <[email protected]>
|
Landed in 7b6c428...997e97d |
Refs: #21076 PR-URL: #21080 Reviewed-By: Tobias Nießen <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Anatoli Papirovski <[email protected]> Reviewed-By: Сковорода Никита Андреевич <[email protected]> Reviewed-By: Yuta Hiroto <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Tiancheng "Timothy" Gu <[email protected]> Reviewed-By: Luigi Pinca <[email protected]>
It transpires that the extra bookkeeping in debug builds sometimes makes the increase in RSS go _just_ over the 5 MB limit, by fewer than 100 kB. Double the limit so we hopefully don't run into it any time again soon. The memory leak it tests for was one where RSS grew by hundreds of megabytes over the lifetime of the test; 5 vs. 10 MB is insignificant. Fixes: #21076 PR-URL: #21080 Refs: #21076 Reviewed-By: Tobias Nießen <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Anatoli Papirovski <[email protected]> Reviewed-By: Сковорода Никита Андреевич <[email protected]> Reviewed-By: Yuta Hiroto <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Tiancheng "Timothy" Gu <[email protected]> Reviewed-By: Luigi Pinca <[email protected]>
Refs: #21076
To help investigate said bug.
CI: https://ci.nodejs.org/job/node-test-pull-request/15212/