ENOMEM with exec/spawn - child process tries to reserve as much mem as parent #25382

zbjornson · 2019-01-07T23:10:29Z

Version: v8.12.0 64-bit
Platform: Linux 4.15.0-1026 (Ubuntu 16.04)
Subsystem: child_process

A tiny exec call when Node.js is using at least half of the otherwise-available memory causes spawn ENOMEM:

// If your system has more than 4 GB of mem, repeat these two lines until >50% of memory is used:
let x = Buffer.allocUnsafe(2e9);
x.fill(2); // virtual -> reserved
// Causes ENOMEM even if there's >1 GB of available memory:
require("child_process").exec("pwd", console.log)

# (strace)
[pid  3017] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7ffae3ea7a10) = -1 ENOMEM (Cannot allocate memory)

Apparently this is a common problem when using fork()/clone(), ~~and using posix_spawn(3) avoids it~~.

Good write up of the issue: https://www.oracle.com/technetwork/server-storage/solaris10/subprocess-136439.html
Related glibc bug discussion from JDK devs working on same issue https://sourceware.org/bugzilla/show_bug.cgi?id=10311
JDK bug that goes with above https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6850720

I don't see any upstream issues in libuv for this.

The text was updated successfully, but these errors were encountered:

zbjornson · 2019-01-07T23:12:14Z

This could be the underlying cause of this very popular SO issue for some users: https://stackoverflow.com/questions/26193654/node-js-catch-enomem-error-thrown-after-spawn

mscdex · 2019-01-08T01:05:04Z

I think something like this would be handled/solved at the libuv level, not node. The libuv issue tracker is here.

vtjnash · 2019-01-08T04:12:44Z

Duplicate of ~~#14917~~?

zbjornson · 2019-01-08T05:05:40Z

Not really a dupe. Both are caused by using fork(), but different symptoms (ENOMEM vs. blocked loop), and this issue isn't addressed by MADV_DONTFORK as suggested in #14917. I just tried my repro with a buffer created from madvise(..., MADV_DONTFORK)'ed memory, and the fork() syscall still failed with ENOMEM. Even if the kernel doesn't make the memory available, it apparently still requests the allocation (or goes through some of the same assertions).

Edit May 23, 2019 tried madvise(MADV_DONTFORK) with a 5.1+ kernel and it still causes ENOMEM.

Edit: in Aug 2019 it does seem to work: #25382 (comment)

bnoordhuis · 2019-01-08T10:01:06Z

What does sysctl vm.overcommit_memory print on your system?

As you've probably gathered from the issues linked in libuv/libuv#2133, this is not something that is solvable, easily or at all. vfork() and clone() have their own share of issues.

vtjnash · 2019-01-08T16:40:15Z

@bnoordhuis Can you point me in the right direction for the issues you've encountered with using vfork? I've tried to search the issue tracker and commit history, but seemed hard to discover back that far. I've seen a couple references to cases where it was seen to have issues (such as libc developers arguing about it https://sourceware.org/bugzilla/show_bug.cgi?id=10354), but fork seems to share most of the same limitations (posix also specifies that "Consequently, [after fork] to avoid errors, the child process may only execute async-signal-safe operations until such time as one of the exec functions is called."), but additionally (on linux) has been observed to be slower and to fail in low-memory situations. Additionally on linux, libuv could use clone directly to avoid some of the issues (with the vfork flag, but providing a separate stack). I did find this interesting graph though, which suggests that the problems with fork might be linux-specific: https://github.com/rtomayko/posix-spawn

It's been a couple years, but when I looked into the linux kernel code for fork, my recollection is that it does not respect the overcommit_memory flag(s) and just enforces that process memory < physical memory (plus probably some extra constant fudge factors). I couldn't really say what memory it counts against the limit however (e.g. if it includes MADV_DONTFORK).

bnoordhuis · 2019-01-08T19:10:57Z

The biggest issue for libuv is that vfork() doesn't run pthread_atfork() handlers. That might be surmountable for Node.js but add-ons might be a problem.

It's been a couple years, but when I looked into the linux kernel code for fork, my recollection is that it does not respect the overcommit_memory flag(s)

Linux's overcommit logic has been overhauled several times in the last years. I wouldn't know where exactly it stands now without checking the source (and also of past releases.) :-)

vtjnash · 2019-01-08T20:29:47Z

They aren't run though because they shouldn't often be needed. Although I don't know what Node.js does (or promises to do) in this case. From the rationale of pthread_atfork, running them is necessary to work around design issues with fork arising due to the COW semantics in multi-threaded programs that try to also use "unsafe" functions. But we can audit uv_spawn and know that it only uses the async-safe functions then execv (per fork documentation). Did someone actually need their pthread_atfork handler to run, or is that just a hypothetical concern that someone might try to do something more in their pthread_atfork handler?

OTOH, libuv now does register a pthread_atfork handler which does some extra operations that adds a couple of additional unnecessary syscalls, and thus may increase the cost of launching a process without vfork. I have no idea whether the impact of that code is measurable though.

But if the overcommit-related logic has changed, perhaps it might be faster now too? I see the OP is on an older Ubuntu 16.04. But if the kernel can do fork (almost) as fast and now as vfork and without hitting ENOMEM limits (as it seems may be true on mach, so perhaps possible at the hardware level), that would great! (and would save me from actually needing to ever upstream the vfork support to libuv, haha)

zbjornson · 2019-01-08T20:32:01Z

@bnoordhuis the default overcommit_memory was 0. The OP repro run with the three options:

0 - ENOMEM
1 - okay
2 - ENOMEM (with additional difficulty allocating the buffer in the first place)

Aside from that workaround (possibly viable) or adding swap (not viable), for my specific situation I'm looking into a PR to make https://github.com/googleapis/google-auth-library-nodejs either not exec at all, or exec earlier before the process grows.

bnoordhuis · 2019-01-10T06:13:49Z

Although I don't know what Node.js does (or promises to do) in this case.

@vtjnash There is (or was) at least one shared memory add-on module that uses pthread_atfork() so I don't think it's out of the question that other add-ons exist that, directly or indirectly, rely on the current behavior.

I say "indirectly" because add-on X might depend on shared library Y, which in turn depends on a functioning pthread_atfork(). Since that's impossible to audit for, I'd be uncomfortable changing the default, but, as discussed in libuv/libuv#141, an opt-in should be acceptable.

@zbjornson Thanks for checking! I figured that DWIM mode (vm.overcommit_memory=1) would end up working around this.

It's curious that madvise(MADV_DONTFORK) didn't work and I wonder if that's a kernel bug. It's not the behavior I'd expect at any rate.

tomasdev · 2019-05-29T19:47:31Z

(Random data point: also seeing this on a 2GB server trying to execute imagemin via CLI)

zbjornson · 2019-08-24T18:14:04Z

Hrm, twice previously I tried using MADV_DONTFORK and was still getting ENOMEMs, but looking at the kernel source to figure out why, I saw that it should work. I just tried again and, lo and behold, the ENOMEM goes away. (Maybe I didn't have a page-aligned allocation or maybe I had two advice flags OR'ed together, and wasn't checking the return value? /shrug, and sorry for the wasted cycles thinking about alternate solutions.)

Anyway, getting https://chromium-review.googlesource.com/c/v8/v8/+/1101679 landed would be great. Looks like there's just a typo pending.

(Also, on my system an madvise call only takes ~6 μs.)

gireeshpunathil · 2020-01-04T05:06:03Z

can we push https://chromium-review.googlesource.com/c/v8/v8/+/1101679 forward?

gireeshpunathil · 2020-01-04T05:06:48Z

cc @bnoordhuis

hugodes · 2020-01-30T15:56:13Z

This would explain why we've had some ENOMEM errors on production since switching to Node 12.

Because default old space size was 1.4G on Node 10, 1.4*2=2.8GB was still under our 4GB server's ram.

Since Node 12 removed the old space size limit, the heap size was sometimes over 3GB, so the spawn would require 3*2 =6GB, and our instances -who do not have SWAP-, would just throw an ENOMEM error.

I will keep an eye on this issue !

temaivanoff · 2020-04-14T09:12:32Z

can we push https://chromium-review.googlesource.com/c/v8/v8/+/1101679 forward?

Hi, there is progress on this topic ?

sdrsdr · 2020-04-30T16:45:33Z

From what I've read the cheapest way to safely do it is use vfork+exec. True fork should setup just a memory with most/all pages mapped from the parent but this is not easy/fast task for huge memory hog as most of node process tend to become. So if we're going to exec past fork and if vfork exist in the system why not use it?

tysonclugg · 2020-07-09T01:44:51Z

Stumbled on this issue trying to figure out why child_process.exec and it's peers use the spawn syscall instead of execve:

node/lib/child_process.js

Line 237 in f1ae7ea

const child = spawn(file, args, {

Given that the execve syscall should act such that the machine code, data, heap, and stack of the process are replaced by those of the new program, it may be reasonable to expect that calling child_process.exec should not result in an out of memory condition. The calling process machine code, data, heap and stacks should be freed.

I'm not sure why spawn is called rather than exec, especially given that child_process.spawn and others are available. Seems to me that it's a bad idea to have both spawn and exec available, but not following expected semantics. One path forward may be to create a new set of methods such as child_process.overlay which do actually call exec, and make it clear in the docs that exec doesn't follow the typical exec paradigm.

tysonclugg · 2020-07-09T05:39:52Z

Further to my comment above, perhaps it would help to consider that using the exec syscall in preference to spawn is a form of tail call elimination, which is useful to limit memory usage.

Here are the results of running the existing benchmark. Note that this optimization helps more for applications with larger heaps, so this is somewhat of an underestimate of the real world performance benefits. ```console $ ./node benchmark/compare.js --runs 15 \ --new ./node \ --old ~/node-v20/out/Release/node \ --filter params child_process > cpr $ node-benchmark-compare cpr confidence improvement (***) methodName='exec' n=1000 *** 60.84 % ±5.43% methodName='execFile' n=1000 *** 53.72 % ±3.33% methodName='execFileSync' n=1000 *** 9.10 % ±0.84% methodName='execSync' n=1000 *** 10.44 % ±0.97% methodName='spawn' n=1000 *** 53.10 % ±2.90% methodName='spawnSync' n=1000 *** 8.64 % ±1.22% 0.01 false positives, when considering a 0.1% risk acceptance (***) ``` Fixes: nodejs#25382 Fixes: nodejs#14917 Refs: nodejs/performance#93 Refs: nodejs/performance#89

Speed up child_process.spawn by enabling the new V8 build flag which makes fork/exec faster. Here are the results of running the existing benchmark. Note that this optimization helps more for applications with larger heaps, so this is somewhat of an underestimate of the real world performance benefits. ```console $ ./node benchmark/compare.js --runs 15 \ --new ./node \ --old ~/node-v20/out/Release/node \ --filter params child_process > cpr $ node-benchmark-compare cpr confidence improvement (***) methodName='exec' n=1000 *** 60.84 % ±5.43% methodName='execFile' n=1000 *** 53.72 % ±3.33% methodName='execFileSync' n=1000 *** 9.10 % ±0.84% methodName='execSync' n=1000 *** 10.44 % ±0.97% methodName='spawn' n=1000 *** 53.10 % ±2.90% methodName='spawnSync' n=1000 *** 8.64 % ±1.22% 0.01 false positives, when considering a 0.1% risk acceptance (***) ``` Fixes: nodejs#25382 Fixes: nodejs#14917 Refs: nodejs/performance#93 Refs: nodejs/performance#89

Speed up child_process.spawn by enabling the new V8 build flag which makes fork/exec faster. Here are the results of running the existing benchmark. Note that this optimization helps more for applications with larger heaps, so this is somewhat of an underestimate of the real world performance benefits. ```console $ ./node benchmark/compare.js --runs 15 \ --new ./node \ --old ~/node-v20/out/Release/node \ --filter params child_process > cpr $ node-benchmark-compare cpr confidence improvement (***) methodName='exec' n=1000 *** 60.84 % ±5.43% methodName='execFile' n=1000 *** 53.72 % ±3.33% methodName='execFileSync' n=1000 *** 9.10 % ±0.84% methodName='execSync' n=1000 *** 10.44 % ±0.97% methodName='spawn' n=1000 *** 53.10 % ±2.90% methodName='spawnSync' n=1000 *** 8.64 % ±1.22% 0.01 false positives, when considering a 0.1% risk acceptance (***) ``` Fixes: #25382 Fixes: #14917 Refs: nodejs/performance#93 Refs: nodejs/performance#89 PR-URL: #48523 Refs: v8/v8@1a782f6 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Juan José Arboleda <[email protected]> Reviewed-By: Debadree Chatterjee <[email protected]>

Original commit message: [base] add build flag to use MADV_DONTFORK Embedders like Node.js and Electron expose fork(2)/execve(2) to their users. Unfortunately when the V8 heap is very large, these APIs become rather slow on Linux, due to the kernel needing to do all the bookkeeping for the forked process (in clone's dup_mmap and execve's exec_mmap). Of course, this is useless because the forked child thread will never actually need to access the V8 heap. Add a new build flag v8_enable_private_mapping_fork_optimization which marks all pages allocated by OS::Allocate as MADV_DONTFORK. This improves the performance of Node.js's fork/execve combination by 10x on a 600 MB heap. Fixed: v8:7381 Change-Id: Ib649f774d4a932b41886313ce89acc369923699d Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4602858 Commit-Queue: Michael Lippautz <[email protected]> Reviewed-by: Michael Lippautz <[email protected]> Cr-Commit-Position: refs/heads/main@{#88447} Refs: v8/v8@1a782f6 PR-URL: #48523 Fixes: #25382 Fixes: #14917 Refs: nodejs/performance#93 Refs: nodejs/performance#89 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Juan José Arboleda <[email protected]> Reviewed-By: Debadree Chatterjee <[email protected]>

Speed up child_process.spawn by enabling the new V8 build flag which makes fork/exec faster. Here are the results of running the existing benchmark. Note that this optimization helps more for applications with larger heaps, so this is somewhat of an underestimate of the real world performance benefits. ```console $ ./node benchmark/compare.js --runs 15 \ --new ./node \ --old ~/node-v20/out/Release/node \ --filter params child_process > cpr $ node-benchmark-compare cpr confidence improvement (***) methodName='exec' n=1000 *** 60.84 % ±5.43% methodName='execFile' n=1000 *** 53.72 % ±3.33% methodName='execFileSync' n=1000 *** 9.10 % ±0.84% methodName='execSync' n=1000 *** 10.44 % ±0.97% methodName='spawn' n=1000 *** 53.10 % ±2.90% methodName='spawnSync' n=1000 *** 8.64 % ±1.22% 0.01 false positives, when considering a 0.1% risk acceptance (***) ``` Fixes: #25382 Fixes: #14917 Refs: nodejs/performance#93 Refs: nodejs/performance#89 PR-URL: #48523 Refs: v8/v8@1a782f6 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Juan José Arboleda <[email protected]> Reviewed-By: Debadree Chatterjee <[email protected]>

Original commit message: [base] add build flag to use MADV_DONTFORK Embedders like Node.js and Electron expose fork(2)/execve(2) to their users. Unfortunately when the V8 heap is very large, these APIs become rather slow on Linux, due to the kernel needing to do all the bookkeeping for the forked process (in clone's dup_mmap and execve's exec_mmap). Of course, this is useless because the forked child thread will never actually need to access the V8 heap. Add a new build flag v8_enable_private_mapping_fork_optimization which marks all pages allocated by OS::Allocate as MADV_DONTFORK. This improves the performance of Node.js's fork/execve combination by 10x on a 600 MB heap. Fixed: v8:7381 Change-Id: Ib649f774d4a932b41886313ce89acc369923699d Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4602858 Commit-Queue: Michael Lippautz <[email protected]> Reviewed-by: Michael Lippautz <[email protected]> Cr-Commit-Position: refs/heads/main@{#88447} Refs: v8/v8@1a782f6 PR-URL: nodejs#48523 Fixes: nodejs#25382 Fixes: nodejs#14917 Refs: nodejs/performance#93 Refs: nodejs/performance#89 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Juan José Arboleda <[email protected]> Reviewed-By: Debadree Chatterjee <[email protected]>

Speed up child_process.spawn by enabling the new V8 build flag which makes fork/exec faster. Here are the results of running the existing benchmark. Note that this optimization helps more for applications with larger heaps, so this is somewhat of an underestimate of the real world performance benefits. ```console $ ./node benchmark/compare.js --runs 15 \ --new ./node \ --old ~/node-v20/out/Release/node \ --filter params child_process > cpr $ node-benchmark-compare cpr confidence improvement (***) methodName='exec' n=1000 *** 60.84 % ±5.43% methodName='execFile' n=1000 *** 53.72 % ±3.33% methodName='execFileSync' n=1000 *** 9.10 % ±0.84% methodName='execSync' n=1000 *** 10.44 % ±0.97% methodName='spawn' n=1000 *** 53.10 % ±2.90% methodName='spawnSync' n=1000 *** 8.64 % ±1.22% 0.01 false positives, when considering a 0.1% risk acceptance (***) ``` Fixes: nodejs#25382 Fixes: nodejs#14917 Refs: nodejs/performance#93 Refs: nodejs/performance#89 PR-URL: nodejs#48523 Refs: v8/v8@1a782f6 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Juan José Arboleda <[email protected]> Reviewed-By: Debadree Chatterjee <[email protected]>

Original commit message: [base] add build flag to use MADV_DONTFORK Embedders like Node.js and Electron expose fork(2)/execve(2) to their users. Unfortunately when the V8 heap is very large, these APIs become rather slow on Linux, due to the kernel needing to do all the bookkeeping for the forked process (in clone's dup_mmap and execve's exec_mmap). Of course, this is useless because the forked child thread will never actually need to access the V8 heap. Add a new build flag v8_enable_private_mapping_fork_optimization which marks all pages allocated by OS::Allocate as MADV_DONTFORK. This improves the performance of Node.js's fork/execve combination by 10x on a 600 MB heap. Fixed: v8:7381 Change-Id: Ib649f774d4a932b41886313ce89acc369923699d Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4602858 Commit-Queue: Michael Lippautz <[email protected]> Reviewed-by: Michael Lippautz <[email protected]> Cr-Commit-Position: refs/heads/main@{#88447} Refs: v8/v8@1a782f6 PR-URL: nodejs#48523 Fixes: nodejs#25382 Fixes: nodejs#14917 Refs: nodejs/performance#93 Refs: nodejs/performance#89 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Juan José Arboleda <[email protected]> Reviewed-By: Debadree Chatterjee <[email protected]>

Speed up child_process.spawn by enabling the new V8 build flag which makes fork/exec faster. Here are the results of running the existing benchmark. Note that this optimization helps more for applications with larger heaps, so this is somewhat of an underestimate of the real world performance benefits. ```console $ ./node benchmark/compare.js --runs 15 \ --new ./node \ --old ~/node-v20/out/Release/node \ --filter params child_process > cpr $ node-benchmark-compare cpr confidence improvement (***) methodName='exec' n=1000 *** 60.84 % ±5.43% methodName='execFile' n=1000 *** 53.72 % ±3.33% methodName='execFileSync' n=1000 *** 9.10 % ±0.84% methodName='execSync' n=1000 *** 10.44 % ±0.97% methodName='spawn' n=1000 *** 53.10 % ±2.90% methodName='spawnSync' n=1000 *** 8.64 % ±1.22% 0.01 false positives, when considering a 0.1% risk acceptance (***) ``` Fixes: nodejs#25382 Fixes: nodejs#14917 Refs: nodejs/performance#93 Refs: nodejs/performance#89 PR-URL: nodejs#48523 Refs: v8/v8@1a782f6 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Juan José Arboleda <[email protected]> Reviewed-By: Debadree Chatterjee <[email protected]>

Original commit message: [base] add build flag to use MADV_DONTFORK Embedders like Node.js and Electron expose fork(2)/execve(2) to their users. Unfortunately when the V8 heap is very large, these APIs become rather slow on Linux, due to the kernel needing to do all the bookkeeping for the forked process (in clone's dup_mmap and execve's exec_mmap). Of course, this is useless because the forked child thread will never actually need to access the V8 heap. Add a new build flag v8_enable_private_mapping_fork_optimization which marks all pages allocated by OS::Allocate as MADV_DONTFORK. This improves the performance of Node.js's fork/execve combination by 10x on a 600 MB heap. Fixed: v8:7381 Change-Id: Ib649f774d4a932b41886313ce89acc369923699d Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4602858 Commit-Queue: Michael Lippautz <[email protected]> Reviewed-by: Michael Lippautz <[email protected]> Cr-Commit-Position: refs/heads/main@{#88447} Refs: v8/v8@1a782f6 PR-URL: nodejs#48523 Fixes: nodejs#25382 Fixes: nodejs#14917 Refs: nodejs/performance#93 Refs: nodejs/performance#89 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Juan José Arboleda <[email protected]> Reviewed-By: Debadree Chatterjee <[email protected]>

Speed up child_process.spawn by enabling the new V8 build flag which makes fork/exec faster. Here are the results of running the existing benchmark. Note that this optimization helps more for applications with larger heaps, so this is somewhat of an underestimate of the real world performance benefits. ```console $ ./node benchmark/compare.js --runs 15 \ --new ./node \ --old ~/node-v20/out/Release/node \ --filter params child_process > cpr $ node-benchmark-compare cpr confidence improvement (***) methodName='exec' n=1000 *** 60.84 % ±5.43% methodName='execFile' n=1000 *** 53.72 % ±3.33% methodName='execFileSync' n=1000 *** 9.10 % ±0.84% methodName='execSync' n=1000 *** 10.44 % ±0.97% methodName='spawn' n=1000 *** 53.10 % ±2.90% methodName='spawnSync' n=1000 *** 8.64 % ±1.22% 0.01 false positives, when considering a 0.1% risk acceptance (***) ``` Fixes: nodejs#25382 Fixes: nodejs#14917 Refs: nodejs/performance#93 Refs: nodejs/performance#89 PR-URL: nodejs#48523 Refs: v8/v8@1a782f6 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Juan José Arboleda <[email protected]> Reviewed-By: Debadree Chatterjee <[email protected]>

Original commit message: [base] add build flag to use MADV_DONTFORK Embedders like Node.js and Electron expose fork(2)/execve(2) to their users. Unfortunately when the V8 heap is very large, these APIs become rather slow on Linux, due to the kernel needing to do all the bookkeeping for the forked process (in clone's dup_mmap and execve's exec_mmap). Of course, this is useless because the forked child thread will never actually need to access the V8 heap. Add a new build flag v8_enable_private_mapping_fork_optimization which marks all pages allocated by OS::Allocate as MADV_DONTFORK. This improves the performance of Node.js's fork/execve combination by 10x on a 600 MB heap. Fixed: v8:7381 Change-Id: Ib649f774d4a932b41886313ce89acc369923699d Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4602858 Commit-Queue: Michael Lippautz <[email protected]> Reviewed-by: Michael Lippautz <[email protected]> Cr-Commit-Position: refs/heads/main@{#88447} Refs: v8/v8@1a782f6 PR-URL: #48523 Backport-PR-URL: #50098 Fixes: #25382 Fixes: #14917 Refs: nodejs/performance#93 Refs: nodejs/performance#89 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Juan José Arboleda <[email protected]> Reviewed-By: Debadree Chatterjee <[email protected]>

Speed up child_process.spawn by enabling the new V8 build flag which makes fork/exec faster. Here are the results of running the existing benchmark. Note that this optimization helps more for applications with larger heaps, so this is somewhat of an underestimate of the real world performance benefits. ```console $ ./node benchmark/compare.js --runs 15 \ --new ./node \ --old ~/node-v20/out/Release/node \ --filter params child_process > cpr $ node-benchmark-compare cpr confidence improvement (***) methodName='exec' n=1000 *** 60.84 % ±5.43% methodName='execFile' n=1000 *** 53.72 % ±3.33% methodName='execFileSync' n=1000 *** 9.10 % ±0.84% methodName='execSync' n=1000 *** 10.44 % ±0.97% methodName='spawn' n=1000 *** 53.10 % ±2.90% methodName='spawnSync' n=1000 *** 8.64 % ±1.22% 0.01 false positives, when considering a 0.1% risk acceptance (***) ``` Fixes: #25382 Fixes: #14917 Refs: nodejs/performance#93 Refs: nodejs/performance#89 PR-URL: #48523 Backport-PR-URL: #50098 Refs: v8/v8@1a782f6 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Juan José Arboleda <[email protected]> Reviewed-By: Debadree Chatterjee <[email protected]>

Original commit message: [base] add build flag to use MADV_DONTFORK Embedders like Node.js and Electron expose fork(2)/execve(2) to their users. Unfortunately when the V8 heap is very large, these APIs become rather slow on Linux, due to the kernel needing to do all the bookkeeping for the forked process (in clone's dup_mmap and execve's exec_mmap). Of course, this is useless because the forked child thread will never actually need to access the V8 heap. Add a new build flag v8_enable_private_mapping_fork_optimization which marks all pages allocated by OS::Allocate as MADV_DONTFORK. This improves the performance of Node.js's fork/execve combination by 10x on a 600 MB heap. Fixed: v8:7381 Change-Id: Ib649f774d4a932b41886313ce89acc369923699d Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4602858 Commit-Queue: Michael Lippautz <[email protected]> Reviewed-by: Michael Lippautz <[email protected]> Cr-Commit-Position: refs/heads/main@{#88447} Refs: v8/v8@1a782f6 PR-URL: nodejs/node#48523 Backport-PR-URL: nodejs/node#50098 Fixes: nodejs/node#25382 Fixes: nodejs/node#14917 Refs: nodejs/performance#93 Refs: nodejs/performance#89 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Juan José Arboleda <[email protected]> Reviewed-By: Debadree Chatterjee <[email protected]>

Speed up child_process.spawn by enabling the new V8 build flag which makes fork/exec faster. Here are the results of running the existing benchmark. Note that this optimization helps more for applications with larger heaps, so this is somewhat of an underestimate of the real world performance benefits. ```console $ ./node benchmark/compare.js --runs 15 \ --new ./node \ --old ~/node-v20/out/Release/node \ --filter params child_process > cpr $ node-benchmark-compare cpr confidence improvement (***) methodName='exec' n=1000 *** 60.84 % ±5.43% methodName='execFile' n=1000 *** 53.72 % ±3.33% methodName='execFileSync' n=1000 *** 9.10 % ±0.84% methodName='execSync' n=1000 *** 10.44 % ±0.97% methodName='spawn' n=1000 *** 53.10 % ±2.90% methodName='spawnSync' n=1000 *** 8.64 % ±1.22% 0.01 false positives, when considering a 0.1% risk acceptance (***) ``` Fixes: nodejs/node#25382 Fixes: nodejs/node#14917 Refs: nodejs/performance#93 Refs: nodejs/performance#89 PR-URL: nodejs/node#48523 Backport-PR-URL: nodejs/node#50098 Refs: v8/v8@1a782f6 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Juan José Arboleda <[email protected]> Reviewed-By: Debadree Chatterjee <[email protected]>

Original commit message: [base] add build flag to use MADV_DONTFORK Embedders like Node.js and Electron expose fork(2)/execve(2) to their users. Unfortunately when the V8 heap is very large, these APIs become rather slow on Linux, due to the kernel needing to do all the bookkeeping for the forked process (in clone's dup_mmap and execve's exec_mmap). Of course, this is useless because the forked child thread will never actually need to access the V8 heap. Add a new build flag v8_enable_private_mapping_fork_optimization which marks all pages allocated by OS::Allocate as MADV_DONTFORK. This improves the performance of Node.js's fork/execve combination by 10x on a 600 MB heap. Fixed: v8:7381 Change-Id: Ib649f774d4a932b41886313ce89acc369923699d Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/4602858 Commit-Queue: Michael Lippautz <[email protected]> Reviewed-by: Michael Lippautz <[email protected]> Cr-Commit-Position: refs/heads/main@{#88447} Refs: v8/v8@1a782f6 PR-URL: nodejs/node#48523 Backport-PR-URL: nodejs/node#50098 Fixes: nodejs/node#25382 Fixes: nodejs/node#14917 Refs: nodejs/performance#93 Refs: nodejs/performance#89 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Juan José Arboleda <[email protected]> Reviewed-By: Debadree Chatterjee <[email protected]>

Speed up child_process.spawn by enabling the new V8 build flag which makes fork/exec faster. Here are the results of running the existing benchmark. Note that this optimization helps more for applications with larger heaps, so this is somewhat of an underestimate of the real world performance benefits. ```console $ ./node benchmark/compare.js --runs 15 \ --new ./node \ --old ~/node-v20/out/Release/node \ --filter params child_process > cpr $ node-benchmark-compare cpr confidence improvement (***) methodName='exec' n=1000 *** 60.84 % ±5.43% methodName='execFile' n=1000 *** 53.72 % ±3.33% methodName='execFileSync' n=1000 *** 9.10 % ±0.84% methodName='execSync' n=1000 *** 10.44 % ±0.97% methodName='spawn' n=1000 *** 53.10 % ±2.90% methodName='spawnSync' n=1000 *** 8.64 % ±1.22% 0.01 false positives, when considering a 0.1% risk acceptance (***) ``` Fixes: nodejs/node#25382 Fixes: nodejs/node#14917 Refs: nodejs/performance#93 Refs: nodejs/performance#89 PR-URL: nodejs/node#48523 Backport-PR-URL: nodejs/node#50098 Refs: v8/v8@1a782f6 Reviewed-By: Yagiz Nizipli <[email protected]> Reviewed-By: Juan José Arboleda <[email protected]> Reviewed-By: Debadree Chatterjee <[email protected]>

zbjornson mentioned this issue Jan 8, 2019

fork() in uv_spawn tries to reserve as much memory as parent libuv/libuv#2133

Open

bnoordhuis added child_process Issues and PRs related to the child_process subsystem. memory Issues and PRs related to the memory management or memory footprint. labels Jan 8, 2019

bcoe mentioned this issue Jan 14, 2019

child_process: child_process.spawn ENOMEM on Windows #25484

Closed

dawnmist mentioned this issue May 3, 2019

Builds fail in CI timarney/react-app-rewired#391

Closed

zbjornson mentioned this issue May 28, 2019

GCLOUD_PROJECT is deprecared in favor of GOOGLE_CLOUD_PROJECT + avoid using it at all GoogleCloudPlatform/nodejs-getting-started#251

Closed

tomasdev mentioned this issue May 29, 2019

ENOMEM on server with 2GB of RAM imagemin/imagemin#321

Open

rphillips mentioned this issue Jul 15, 2019

Bump lit/luvi versions to latest for testing luvit/luvit#1063

Closed

ISNIT0 mentioned this issue Aug 27, 2019

WPEN is not scrapeable anymore in 16GB of RAM openzim/mwoffliner#948

Closed

hexsprite mentioned this issue May 9, 2020

[1.10.2] ENOMEM building in CI meteor/meteor#11050

Closed

jasnell added the help wanted Issues that need assistance from volunteers or PRs that need help to proceed. label Jun 26, 2020

LaurensUP mentioned this issue Sep 9, 2021

[Bug?]: workspaces foreach using too much memory yarnpkg/berry#3395

Open

1 task

kvakil mentioned this issue Jun 22, 2023

child_process: improve spawn performance on Linux #48523

Closed

nodejs-github-bot closed this as completed in ce4102e Jun 24, 2023

tysonclugg mentioned this issue Jun 11, 2024

tsup build failing with ERR_WORKER_OUT_OF_MEMORY egoist/tsup#920

Open

dvail mentioned this issue Aug 29, 2024

fix(ui): Disable npm postinstall scripts on konflux builds stackrox/stackrox#12506

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENOMEM with exec/spawn - child process tries to reserve as much mem as parent #25382

ENOMEM with exec/spawn - child process tries to reserve as much mem as parent #25382

zbjornson commented Jan 7, 2019 •

edited

Loading

zbjornson commented Jan 7, 2019

mscdex commented Jan 8, 2019

vtjnash commented Jan 8, 2019

zbjornson commented Jan 8, 2019 •

edited

Loading

bnoordhuis commented Jan 8, 2019

vtjnash commented Jan 8, 2019

bnoordhuis commented Jan 8, 2019

vtjnash commented Jan 8, 2019

zbjornson commented Jan 8, 2019

bnoordhuis commented Jan 10, 2019

tomasdev commented May 29, 2019

zbjornson commented Aug 24, 2019

gireeshpunathil commented Jan 4, 2020

gireeshpunathil commented Jan 4, 2020

hugodes commented Jan 30, 2020 •

edited

Loading

temaivanoff commented Apr 14, 2020 •

edited

Loading

sdrsdr commented Apr 30, 2020

tysonclugg commented Jul 9, 2020

tysonclugg commented Jul 9, 2020

ENOMEM with exec/spawn - child process tries to reserve as much mem as parent #25382

ENOMEM with exec/spawn - child process tries to reserve as much mem as parent #25382

Comments

zbjornson commented Jan 7, 2019 • edited Loading

zbjornson commented Jan 7, 2019

mscdex commented Jan 8, 2019

vtjnash commented Jan 8, 2019

zbjornson commented Jan 8, 2019 • edited Loading

bnoordhuis commented Jan 8, 2019

vtjnash commented Jan 8, 2019

bnoordhuis commented Jan 8, 2019

vtjnash commented Jan 8, 2019

zbjornson commented Jan 8, 2019

bnoordhuis commented Jan 10, 2019

tomasdev commented May 29, 2019

zbjornson commented Aug 24, 2019

gireeshpunathil commented Jan 4, 2020

gireeshpunathil commented Jan 4, 2020

hugodes commented Jan 30, 2020 • edited Loading

temaivanoff commented Apr 14, 2020 • edited Loading

sdrsdr commented Apr 30, 2020

tysonclugg commented Jul 9, 2020

tysonclugg commented Jul 9, 2020

zbjornson commented Jan 7, 2019 •

edited

Loading

zbjornson commented Jan 8, 2019 •

edited

Loading

hugodes commented Jan 30, 2020 •

edited

Loading

temaivanoff commented Apr 14, 2020 •

edited

Loading