Process aborts with 'out of memory' when using 2.0.0

[nick]

• Operating System: OSX 10.14.6
• Node Version: 10.16.0
• NPM Version: 6.9.0
• webpack Version: 4.39.3
• terser-webpack-plugin Version: 2.0.0

Expected Behavior

Process does not abort

Actual Behavior

$ NODE_ENV=production ./node_modules/.bin/webpack --loglevel notice

<--- Last few GCs --->

[84294:0x102843000]    55749 ms: Mark-sweep 1312.4 (1444.3) -> 1305.0 (1446.3) MB, 622.0 / 0.0 ms  (average mu = 0.099, current mu = 0.040) allocation failure scavenge might not succeed
[84294:0x102843000]    56388 ms: Mark-sweep 1315.7 (1446.3) -> 1308.6 (1448.8) MB, 613.1 / 0.0 ms  (average mu = 0.070, current mu = 0.040) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x3aa9ebf5be3d]
Security context: 0x37c8eea1e6e9 <JSObject>
    1: /* anonymous */(aka /* anonymous */) [0x37c895d8d969] [/Users/nick/Projects/origin/node_modules/terser-webpack-plugin/node_modules/webpack-sources/lib/applySourceMap.js:~58] [pc=0x3aa9ed4fe2c4](this=0x37c83a5026f1 <undefined>,chunk=0x37c803a7d979 <String[14]: createElement(>,middleMapping=0x37c8171046d9 <Object map = 0x37c835e45c79>)
    2: SourceNode...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x10003cf99 node::Abort() [/usr/local/bin/node]
 2: 0x10003d1a3 node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
 3: 0x1001b7835 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 4: 0x100585682 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node]
 5: 0x100588155 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/usr/local/bin/node]
 6: 0x100583fff v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
 7: 0x1005821d4 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
 8: 0x10058ea6c v8::internal::Heap::AllocateRawWithLigthRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/local/bin/node]
 9: 0x10058eaef v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/local/bin/node]
10: 0x10055e434 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/usr/local/bin/node]
11: 0x1007e6714 v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]
12: 0x3aa9ebf5be3d 
13: 0x3aa9ed4fe2c4 
14: 0x3aa9ed4b7d28 
Abort trap: 6

Code

https://github.com/OriginProtocol/origin/blob/master/dapps/marketplace/webpack.config.js

How Do We Reproduce?

git clone https://github.com/OriginProtocol/origin.git
cd origin
# Update dapps/marketplace/package.json to use v2.0.0 of terser plugin
yarn
cd dapps/marketplace
yarn build

[alexander-akait]

Thanks for issue, investigate

[alexander-akait]

Just information, change this line to:

"build:js": "NODE_ENV=production webpack --progress --loglevel notice",

and see what happens.

Error appears on source map generation step:

93% after chunk asset optimization SourceMapDevToolPlugin app.9949447c.js generate SourceMap`

Use sourceMap: false fix problem, increase memory for node also fix problem. I keep investigation.

[alexander-akait]

Looks terser leaking

[alexander-akait]

Why problem appears:
Here we change logic 8b88b39

Before for only one file we run new thread for uglification, now we use same thread for this. Removing tasks.length > 1 from https://github.com/webpack-contrib/terser-webpack-plugin/blob/master/src/TaskRunner.js#L48 fix problem. Why we do this - try to reduce memory and cpu usage when you have only one file (creating new thread takes more memory and increase cpu loading).

But another problem occurred - memory leak. terser something doesn’t clear therefore the current thread began to consume more memory.

Simple investigation, i used this code:

const used = process.memoryUsage().heapUsed / 1024 / 1024;
console.log(`The script uses approximately ${Math.round(used * 100) / 100} MB`);

and put this lines after https://github.com/webpack-contrib/terser-webpack-plugin/blob/master/src/index.js#L272

New version (we don't create new thread for one file):

The script uses approximately 1016.37 MB

Old version (new thread for one file):

The script uses approximately 362.56 MB

Around 700 MB leaking

/cc @fabiosantoscode

[alexander-akait]

/cc @nick also cache is not working with your code, because you always generate something new in code, don't use:

BUILD_TIMESTAMP: +new Date()

Caches is not working in your app, long cache term also

[alexander-akait]

@nick try to use https://github.com/webpack-contrib/terser-webpack-plugin/releases/tag/v2.0.1, anyway we need investigate why terser consume a lot of memory

Let's wait @fabiosantoscode answer

[nick]

Thanks for looking this so quickly, @evilebottnawi - and thanks for the tip on the cache issue 😊

Can confirm 2.0.1 works as before. Feel free to close unless you want to keep it open for the memory issue.

[fabiosantoscode]

Interesting. I'll try to come up with the exact tag where this started to happen.

[fabiosantoscode]

But I kind of have something else I have to fix first, though.

[alexander-akait]

@fabiosantoscode 👍 yep, feel free to ping me i can help you with memory leak searching

[fabiosantoscode]

I tried creating a file that attempts to encrypt a lot of things to try and create a memory leak, but I can't reproduce this at all. I looked into the commit that fixes this. Isn't this just a 2.0.0 specific problem?

[alexander-akait]

@fabiosantoscode no, we just run terser in separate thread instead using currently, but looks terser have memory leak, because no problem in parallel mode, but problem when you use only one thread

[fabiosantoscode]

Can this be reproduced with the default options? The Terser tests aren't leaking memory at all.

[eliseumds]

In my experience, there hasn't been any significant difference in memory usage. Both V1 and V2 are hovering around 500MB of memory (and same compilation time) for one of our smaller apps. Measured with NODE_ENV=production time -l ... on MacOS. The dependencies are basically the same (Node, Webpack, etc).

[alexander-akait]

@eliseumds it is expected because terser still consume many memory, need search way to optimize terser and decrease uglify time

[rrelmy]

Build time more than doubled from 6 minutes to 14 minutes in one of our bigger projects by just updating to version 2 of this dependency.

[JoshRobertson]

Our build time also doubled when upgrading to version 2

[alexander-akait]

@rrelmy @JoshRobertson sorry but your comments is not helpful. Please provide information what version you use before, also we have breaking change and generate source maps bsed on devtool value, so now you can generate source map and it is increase build time

[korya]

Same story here. After we upgraded terser-webpack-plugin from v1.4.1 to v2.2.1 the build process started to run out of memory. As a result, our CI builds started to fail. We have some big chunks though: 5 over 1MB, 2 over 2MB. Degrading back to v1.4.1 resolves the problem. Turning off parallel build resolves the problem as well.

The error that we get (it is reported 3 times in a row):

ERROR in c/92f867d3e514eb399b9d.js from Terser
Error: Call retries were exceeded
    at ChildProcessWorker.initialize (/home/circleci/workdir/project/node_modules/jest-worker/build/workers/ChildProcessWorker.js:193:21)
    at ChildProcessWorker.onExit (/home/circleci/workdir/project/node_modules/jest-worker/build/workers/ChildProcessWorker.js:263:12)
    at ChildProcess.emit (events.js:210:5)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:272:12)

Below are the results summarized in a table:

version	parallel build	outcome
v1.4.1	Y	OK
v2.2.1	Y	FAILURE: runs out of memory
v2.2.1	N	OK

Note: in all 3 cases above everything else remained unchanged, only 2 variables changed: the version and the parallel option of the plugin.

[alexander-akait]

@korya can you create reproducible test repo?

[korya]

@evilebottnawi Sorry, I will probably won't able to do it. Our repository is private, and coming up with an artificial example to reproduce this issue would be a nightmare.

It seems that the memory consumption of the plugin just grew up. The v1.4.1 is OK, but it is probably very close to hitting the memory limit. The v2.2.1's memory consumption grew up (maybe a lot, but maybe just a bit) and it started to consistently hit the limit. This is my very surface analysis of our problem. I would even call it an intuition rather than analysis. I also noticed that the number of transitive dependencies of the plugin grew up considerably in v2.2.1.

Again these are just my thoughts, I did not perform any deep analysis. I am sorry that I cannot help any more than that. If I will have some time, I will try to gather more info. But I don't believe I can easily come up with a reproducible test.

[alexander-akait]

@korya anyway thanks for feedback, yes new version require more memory:

• now we don't break source maps anymore
• better parallel support

All this requires a little more memory. We have an issue to reduce memory/cpu usage terser/terser#478.

[mateder]

@korya We had the same error and the same behaviour.

The childprocesses immediately return with an error code of 1 and after a number of retries the error message occurs. I turned off the silent option in ChildProcessWorker.js (of jest-worker).
Then i got following error message: error: unknown option '--trace-deprecation'

Our build scripts had --trace-deprecation enabled and somehow the childprocess is not able to handle this option. I just disabled it for now.

Temporarily replace silent: true with silent: false in node_modules\jest-worker\build\workers\ChildProcessWorker.js and check for error messges.

[alexander-akait]

@mateder Can you create minimum reproducible test repo?

[ahmedelgabri]

Same problem started to appear for us & nothing changed other than adding a new package to our build. In our case also, parallel doesn't make much of a difference instead the issue is sourceMaps if set to false everything works fine. Otherwise out of memory errors.

I tried these versions, 1.3.0 (the one we have), 1.4.1 & 2.2.1 all of them fail.

[alexander-akait]

@ahmedelgabri maybe you can create reproducible test repo, i will investigate that

[alexander-akait]

I think I was able to solve this problem, big thanks @cjlarose for inspiration and great idea, honestly I don’t know how I didn’t get to this decision before. Now I run tests to put them here, I think today we release a patch version and close that problem.

I want to warn that this reduces consumption, but for real big projects, for example 5000+ entries, you need increase node memory usage

[alexander-akait]

Release will be today, evening, want to add more tests (increase coverage)

[throttlehead]

Hey guys just want to comment that this parallism is still unstable but it honestly seems very close to the bare metal here.

I'm on a ryzen 5800 and Ubuntu 20.04. If i let it figure out parallel itself, my cpu and ram max out and the process ends up killed. If I use say an env variable and set it to 6, i borderline max memory usage.

If i run this on one of our web nodes which are typically 4 core amazon t3 instances running 18.04, the build seems to work just fine.

All were ran on node v14.17.0 and the latest webpack and babel versions.

I have limited understanding as to what is happening under the hood but I can say that we have many 'entry' apps, around 30+. If I cull these entries just for sanity check purposes it does seem to help but I still max out and die.

There's something going on here with how it forks and utilizes system resources and like I said it may be only affecting my hardware group here.

If anyone is finding there way to this comment here's an example of how I solved my issue:

  let parallel = 4;

  if (typeof process.env.WEBPACK_PARALLELISM !== 'undefined') {
    parallel = parseInt(process.env.WEBPACK_PARALLELISM);
  }

  config.optimization = {
    minimize: true,
    minimizer: [new TerserPlugin({
      parallel: parallel
    })],
  };