Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allows setting of parallelization options #2093

Merged
merged 2 commits into from
Aug 8, 2020

Conversation

masato-hi
Copy link
Contributor

If you set the parallel option of TersorPlugin to true, resource usage issues may occur.
This problem is noticeable when used on containers, and will fail to fork when the number of CPU cores on hardware is high and the memory usage allowed for the container is low.

This is due to the fact that the number of CPU cores count - 1 process is generated when the parallel option of TersorPlugin is true.

This patch makes it possible to limit the number of parallelizations in the following way.

How to write to config/initializers/webpacker.rb

Webpacker::Compiler.env['WEBPACKER_PARALLEL'] = '2'

or how to write it before doing require('@rails/webpacker') in config/webpack/production.js

process.env.WEBPACKER_PARALLEL = '2'

const environment = require('./environment')

I'm sorry if you find something strange because I use Google Translate.

Copy link
Member

@jakeNiemiec jakeNiemiec left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be fixed upstream in: webpack-contrib/terser-webpack-plugin#21 via webpack-contrib/terser-webpack-plugin#90

Can you confirm that this is still an issue on the latest version? (otherwise, it looks 👍)

@masato-hi
Copy link
Contributor Author

masato-hi commented Jun 12, 2019

Thank you for the review.
But this is not a problem in the WSL environment.

TersorPlugin gets cpu count in this line.
https://github.com/webpack-contrib/terser-webpack-plugin/blob/master/src/TaskRunner.js#L22

When isWsl = false and parallel = true, cpus.length -1 is assigned to maxConcurrentWorkers.
https://github.com/webpack-contrib/terser-webpack-plugin/blob/master/src/TaskRunner.js#L25

And the process is forked and executed by the number of maxConcurrentWorkers.
https://github.com/webpack-contrib/terser-webpack-plugin/blob/master/src/TaskRunner.js#L47

For example, if the usable memory is 1GB and the CPU count is 4 cores and consumes 500MB per process, 1.5GB of memory is required in total and the build fails because of insufficient memory.

Copy link
Member

@jakeNiemiec jakeNiemiec left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure if @gauravtiwari wants to pass settings differently, but it looks good from where I stand.

@jakeNiemiec
Copy link
Member

@masato-hi Could you please add documentation for this change?

@gauravtiwari gauravtiwari merged commit ef04f5c into rails:master Aug 8, 2020
@YurySolovyov
Copy link

It seems that if you want to disable parallel completely there is no way to do that currently:

> Number.parseInt('false', 10) || true
true
> Number.parseInt('0', 10) || true
true

Am I missing something?

@masato-hi
Copy link
Contributor Author

@YurySolovyov Please specify '1'.
Number.parseInt ('false') and Number.parseInt ('0') are NaN and 0, respectively, both of which evaluate to false.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants