feat(gatsby): configure physical cores, logical_cores or fixed number #10257

dominicfallows · 2018-12-03T16:15:44Z

Gatsby 2 now utilises multi-core builds using jest-worker. By default Gatsby creates a pool of workers equal to the number of physical cores on your machine, see build-html.js.

In some scenarios it may be appropriate to tell Gatsby to use a different method to calculate the number of worker pools.

For example, if you are running a Cloud server (like AWS EC2) your DevOps engineers may want to control the number of worker pools to improve the efficiency of server resource usage.

The proposal here is to accept an options env var GATSBY_CPU_COUNT to change the method that Gatsby uses to calculate the number of worker pools.

closes #11727

KyleAMathews · 2018-12-03T16:19:56Z

So the scenario is you're running builds on a server with other work also going on and you want to limit the number of cores Gatsby uses?

dominicfallows · 2018-12-03T16:21:24Z

@KyleAMathews either that yes, or the opposite, sometimes we want to allow logical CPU count (more cores). For example, our AWS instances are dedicated to this process, and therefore we want to be able to push them to their limits and speed up our builds.

KyleAMathews · 2018-12-03T16:25:33Z

Hmm ok — we'll have to think about this a bit as we're also adding in the future multi-core support for running GraphQL queries. So a global CPU limit/count could make sense.

Also BTW, if you're site does any image transformations with sharp, it's already using all CPUs as well. Any solution we do should also probably figure out how to limit sharp.

dominicfallows · 2018-12-03T16:32:04Z

Ok, sounds interesting. I'll have a dig around and look for the sharp handling to see how that could work alongside this.

In future, would you see a global CPU limit/count using an env var as I've drafted, or another config option? etc.

paulca99

Looks good to me

…atsb'y cpuCount handling

dominicfallows · 2018-12-03T19:17:22Z

@KyleAMathews ok, so I've made a first attempt at aligning sharp usage with proposed Gatsby cpuCount usage. Along the right lines based on what you were thinking?

KyleAMathews · 2018-12-11T01:12:08Z

Could you post build logs from using Physical vs. Logical CPU counts?

If that generally speeds up builds — we should just change our default to use the logical CPU count.

I'd rather not add config if we don't need to.

Could you try running some of the benchmarks w/ both physical/logical and see how that changes things? https://github.com/gatsbyjs/gatsby/tree/master/benchmarks

docs/docs/multi-core-builds.md

www/src/data/sidebars/doc-links.yaml

docs/docs/multi-core-builds.md

mik-laj · 2019-01-01T03:30:42Z

Should the main file be called cpu-count? I am afraid that this name can be misleading. I am afraid that few people run the program in a multi-CPU environment. However, everyone runs the program in a multi-core environment. The better name will be core-count, but it is still not correct.

Valid name for me is parallelism.

Reference: https://learn-gevent-socketio.readthedocs.io/en/latest/general_concepts.html#what-s-the-difference-between-concurrency-and-parallelism

packages/gatsby/src/utils/cpu-count.js

wardpeet · 2019-02-27T15:09:25Z

pinging @KyleAMathews & @pieh wdyt?

I think this is great, especially on VMs.

KyleAMathews · 2019-02-27T19:53:39Z

Haven't looked at the PR but 👍 to the concept

wardpeet · 2019-02-28T11:27:18Z

@dominicfallows did you have some benchmarks? I'm a bit hesitant to add this, it's opt-in but for example sharp will probably have a negative impact as it probably best to use cores instead of threads as it has so many heavy computations.

I've found an issue at the Parcel repo and it is not moving to logical cores:
parcel-bundler/parcel#1554 (comment)

packages/gatsby/src/utils/cpu-count.js

…cs/cpu-control-in-html-renderer-queue

- change name of util file and function to prevent confusion - Refactor default count response - Sharp functions now back to default cpu core count (physical or 1) - Throw error if we can't calculate logical_core count

dominicfallows · 2019-02-28T16:56:08Z

@mik-laj updated the file name to cpu-core-count.js and helper function to cpuCoreCount which does seem more accurate, thanks :)

dominicfallows · 2019-02-28T17:00:54Z

@KyleAMathews RE: #10257 (comment)
@wardpeet RE: #10257 (comment)

Benchmarks are proving quite difficult to show using the existing benchmark examples. The improvements are so dependant on the infrastructure setup and setup of the app, hence why I went for an 'opt-in' approach.

I could create a new benchmark example though, that would go to show the type of app setup and infrastructure where I'm seeing improvements (cloud containers and instances, for example), would that help?

I also wondered about a different approach, somehow having this calculation as a plugin. It would still require changes to the core code (to allow plugins to override the CPU core count calculation), but would move away from needing an env var.

wardpeet · 2019-03-01T09:12:54Z

I guess you're using this already on AWS? If so maybe share the numbers you're encountering? Or maybe a shared blog post or document where this really shows physical vs logical on the cloud infra 😄

I also wondered about a different approach, somehow having this calculation as a plugin. It would still require changes to the core code (to allow plugins to override the CPU core count calculation), but would move away from needing an env var.

I don't think we want to expose such an API to the world 😄. For now I think env var is good enough and we need to figure out this a bit more for other things as well like experiments so we're probably going to revisit this for Gatsby 3.

packages/gatsby/src/utils/cpu-core-count.js

paulca99 · 2019-03-01T19:38:17Z

Hi, I'm the dev ops tech guy dealing with the codebuild images running this stuff. To be blunt, Doms "Logical cores" change knocked nearly 30% off our AWS Codebuild timings. If it's a concern to you, make it a variable... use logical . vs . use physical.... give users the option.

paulca99 · 2019-03-01T19:47:51Z

it's been 3 months since Dom, discovered this and you're still debating things. if I were you I'd have added a switch, then debated the output later once you had some user results to judge it on.

wardpeet · 2019-03-01T20:54:32Z

I was going to merge this when tests are passing.

@paulca99 Sorry but we have to deal with a lot of PRs and issues. Because of the holidays, it slipped our mind. There is nothing wrong to ask for validation and extra information before merging even if it's opt-in. Extra code could lead to bugs or build errors.

paulca99 · 2019-03-02T08:29:07Z

Sorry @wardpeet I didn't notice there were tests failing.

dominicfallows · 2019-03-03T23:54:53Z

@wardpeet Hey, I've updated a test in the gatsby-plugin-manifest package and tests now look good. Let me know what else I can share and I'll do it asap.

wardpeet

@dominicfallows thanks for creating this PR! I'm going to merge this one but it would be great if you could share some build times when using it.

Thanks for all your patience!

sidharthachatterjee · 2019-03-04T09:52:47Z

Published in

[email protected]
[email protected]
[email protected]

dominicfallows added 2 commits December 3, 2018 16:09

Handle CPUs based on env var

17ca238

Updated documentation

1ab8598

dominicfallows added the type: feature or enhancement label Dec 3, 2018

dominicfallows self-assigned this Dec 3, 2018

dominicfallows requested a review from a team as a code owner December 3, 2018 16:15

dominicfallows requested a review from a team December 3, 2018 16:15

dominicfallows requested a review from a team as a code owner December 3, 2018 16:15

paulca99 approved these changes Dec 3, 2018

View reviewed changes

dominicfallows added 2 commits December 3, 2018 19:10

Create utilty function for cpuCount, align sharp's concurrency with G…

0ed6edc

…atsb'y cpuCount handling

FIx incorrect cpuCount import

a91d9d5

dominicfallows mentioned this pull request Dec 3, 2018

Building large amount of pages (~16k) on Gatsby V2 performance issues #7373

Closed

+ some bugfixes in handling cpuCount

48ab310

Merge branch 'master' into topics/cpu-control-in-html-renderer-queue

33a759d

shannonbux suggested changes Dec 12, 2018

View reviewed changes

docs/docs/multi-core-builds.md Outdated Show resolved Hide resolved

docs/docs/multi-core-builds.md Outdated Show resolved Hide resolved

www/src/data/sidebars/doc-links.yaml Outdated Show resolved Hide resolved

docs/docs/multi-core-builds.md Outdated Show resolved Hide resolved

pieh mentioned this pull request Feb 13, 2019

--max-workers flag for gatsby build #11727

Closed

wardpeet added the status: awaiting reviewer response A pull request that is currently awaiting a reviewer's response label Feb 27, 2019

wardpeet reviewed Feb 27, 2019

View reviewed changes

packages/gatsby/src/utils/cpu-count.js Outdated Show resolved Hide resolved

Merge branch 'master' into topics/cpu-control-in-html-renderer-queue

d58a96d

wardpeet changed the title ~~Topics/CPU control in htm-renderer-queue.js (multi-core builds)~~ feat(gatsby): configure physical cores, logical_cores or fixed number Feb 28, 2019

wardpeet reviewed Feb 28, 2019

View reviewed changes

packages/gatsby/src/utils/cpu-count.js Outdated Show resolved Hide resolved

wardpeet added status: awaiting author response Additional information has been requested from the author and removed status: awaiting reviewer response A pull request that is currently awaiting a reviewer's response labels Feb 28, 2019

dominicfallows added 2 commits February 28, 2019 13:49

Merge branch 'master' of https://github.com/gatsbyjs/gatsby into topi…

a63205b

…cs/cpu-control-in-html-renderer-queue

Punctuation and grammar tweaks, thank you @shannonbux

d67e1c4

dominicfallows requested a review from shannonbux February 28, 2019 16:22

Updates based on feedback

cac1de7

- change name of util file and function to prevent confusion - Refactor default count response - Sharp functions now back to default cpu core count (physical or 1) - Throw error if we can't calculate logical_core count

dominicfallows added 2 commits March 1, 2019 07:51

Update forgotten import module name change

4af3096

Formatting fix

1a3456e

wardpeet reviewed Mar 1, 2019

View reviewed changes

packages/gatsby/src/utils/cpu-core-count.js Outdated Show resolved Hide resolved

Only import cpu counts as we need them

cdc8578

jeffrafter mentioned this pull request Mar 1, 2019

gatsby develop can only serve 5 pages at a time #12225

Closed

Add sharp.concurrency mock to gatsby-plugin-manifest tests

f4c1754

wardpeet approved these changes Mar 4, 2019

View reviewed changes

wardpeet removed the status: awaiting author response Additional information has been requested from the author label Mar 4, 2019

wardpeet merged commit c51440e into gatsbyjs:master Mar 4, 2019

matt-assemble mentioned this pull request Mar 5, 2019

gatsby-plugin-sharp 2.0.24 requires [email protected] but specifies peer dependency of gatsby@^2.0.0 #12307

Closed

DSchau mentioned this pull request Mar 12, 2019

ci: add build-www task (and lightly refactor scripts) #12325

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(gatsby): configure physical cores, logical_cores or fixed number #10257

feat(gatsby): configure physical cores, logical_cores or fixed number #10257

dominicfallows commented Dec 3, 2018 •

edited by pieh

Loading

KyleAMathews commented Dec 3, 2018

dominicfallows commented Dec 3, 2018

KyleAMathews commented Dec 3, 2018

dominicfallows commented Dec 3, 2018

paulca99 left a comment

dominicfallows commented Dec 3, 2018

KyleAMathews commented Dec 11, 2018

mik-laj commented Jan 1, 2019

wardpeet commented Feb 27, 2019

KyleAMathews commented Feb 27, 2019

wardpeet commented Feb 28, 2019

dominicfallows commented Feb 28, 2019

dominicfallows commented Feb 28, 2019 •

edited

Loading

wardpeet commented Mar 1, 2019

paulca99 commented Mar 1, 2019

paulca99 commented Mar 1, 2019

wardpeet commented Mar 1, 2019

paulca99 commented Mar 2, 2019

dominicfallows commented Mar 3, 2019

wardpeet left a comment

sidharthachatterjee commented Mar 4, 2019

feat(gatsby): configure physical cores, logical_cores or fixed number #10257

feat(gatsby): configure physical cores, logical_cores or fixed number #10257

Conversation

dominicfallows commented Dec 3, 2018 • edited by pieh Loading

KyleAMathews commented Dec 3, 2018

dominicfallows commented Dec 3, 2018

KyleAMathews commented Dec 3, 2018

dominicfallows commented Dec 3, 2018

paulca99 left a comment

Choose a reason for hiding this comment

dominicfallows commented Dec 3, 2018

KyleAMathews commented Dec 11, 2018

mik-laj commented Jan 1, 2019

wardpeet commented Feb 27, 2019

KyleAMathews commented Feb 27, 2019

wardpeet commented Feb 28, 2019

dominicfallows commented Feb 28, 2019

dominicfallows commented Feb 28, 2019 • edited Loading

wardpeet commented Mar 1, 2019

paulca99 commented Mar 1, 2019

paulca99 commented Mar 1, 2019

wardpeet commented Mar 1, 2019

paulca99 commented Mar 2, 2019

dominicfallows commented Mar 3, 2019

wardpeet left a comment

Choose a reason for hiding this comment

sidharthachatterjee commented Mar 4, 2019

dominicfallows commented Dec 3, 2018 •

edited by pieh

Loading

dominicfallows commented Feb 28, 2019 •

edited

Loading