Add experimental options for more parallelization in webpack builds

[mknichel]

This PR introduces 2 experimental options for doing more work in the webpack build in parallel instead of in serial. These options may improve the performance of builds at the cost of more memory.

parallelServerAndEdgeCompiles: This option kicks off the builds for both server and edge-server at the same time instead of waiting for each to complete before the next one. In applications that have many server and edge functions, this can increase performance by doing that work in parallel. This can be used with next build or next experimental-compile.

parallelServerBuildTraces: This option starts the server build traces as soon as the server compile completes and runs it in the background while the other compilations are happening. With this option enabled, some unnecessary work may be done since ordinarily the client compilation provides information that can reduce the amount of tracing necessary. However, since it is in parallel with the other work, it may still result in a faster build in total at the cost of more memory. This option is already the default when using next experimental-compile but can now be used when next build is used also.

[ijjk]

Tests Passed

[ijjk]

Stats from current PR

Default Build

General Overall increase ⚠️

	vercel/next.js canary	vercel/next.js mknichel/parallel-edge-server-and-traces	Change
buildDuration	12.9s	12.7s	N/A
buildDurationCached	7s	6.2s	N/A
nodeModulesSize	199 MB	200 MB	⚠️ +11.8 kB
nextStartRea..uration (ms)	429ms	433ms	N/A

Client Bundles (main, webpack)

	vercel/next.js canary	vercel/next.js mknichel/parallel-edge-server-and-traces	Change
193.HASH.js gzip	181 B	182 B	N/A
3f784ff6-HASH.js gzip	53.3 kB	53.3 kB	N/A
433-HASH.js gzip	28.5 kB	28.6 kB	N/A
framework-HASH.js gzip	45.2 kB	45.2 kB	✓
main-app-HASH.js gzip	240 B	242 B	N/A
main-HASH.js gzip	31.8 kB	31.8 kB	N/A
webpack-HASH.js gzip	1.7 kB	1.7 kB	N/A
Overall change	45.2 kB	45.2 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	vercel/next.js mknichel/parallel-edge-server-and-traces	Change
polyfills-HASH.js gzip	31 kB	31 kB	✓
Overall change	31 kB	31 kB	✓

Client Pages

	vercel/next.js canary	vercel/next.js mknichel/parallel-edge-server-and-traces	Change
_app-HASH.js gzip	194 B	195 B	N/A
_error-HASH.js gzip	183 B	181 B	N/A
amp-HASH.js gzip	504 B	502 B	N/A
css-HASH.js gzip	321 B	321 B	✓
dynamic-HASH.js gzip	2.5 kB	2.5 kB	N/A
edge-ssr-HASH.js gzip	255 B	253 B	N/A
head-HASH.js gzip	350 B	349 B	N/A
hooks-HASH.js gzip	369 B	369 B	✓
image-HASH.js gzip	4.28 kB	4.28 kB	N/A
index-HASH.js gzip	255 B	256 B	N/A
link-HASH.js gzip	2.61 kB	2.61 kB	✓
routerDirect..HASH.js gzip	312 B	311 B	N/A
script-HASH.js gzip	385 B	383 B	N/A
withRouter-HASH.js gzip	307 B	308 B	N/A
1afbb74e6ecf..834.css gzip	106 B	106 B	✓
Overall change	3.4 kB	3.4 kB	✓

Client Build Manifests

	vercel/next.js canary	vercel/next.js mknichel/parallel-edge-server-and-traces	Change
_buildManifest.js gzip	483 B	484 B	N/A
Overall change	0 B	0 B	✓

Rendered Page Sizes

	vercel/next.js canary	vercel/next.js mknichel/parallel-edge-server-and-traces	Change
index.html gzip	528 B	527 B	N/A
link.html gzip	540 B	540 B	✓
withRouter.html gzip	523 B	523 B	✓
Overall change	1.06 kB	1.06 kB	✓

Edge SSR bundle Size

	vercel/next.js canary	vercel/next.js mknichel/parallel-edge-server-and-traces	Change
edge-ssr.js gzip	93.8 kB	93.9 kB	N/A
page.js gzip	148 kB	148 kB	N/A
Overall change	0 B	0 B	✓

Middleware size

	vercel/next.js canary	vercel/next.js mknichel/parallel-edge-server-and-traces	Change
middleware-b..fest.js gzip	623 B	623 B	✓
middleware-r..fest.js gzip	151 B	151 B	✓
middleware.js gzip	37.5 kB	37.5 kB	N/A
edge-runtime..pack.js gzip	1.92 kB	1.92 kB	✓
Overall change	2.7 kB	2.7 kB	✓

Next Runtimes

	vercel/next.js canary	vercel/next.js mknichel/parallel-edge-server-and-traces	Change
app-page-exp...dev.js gzip	169 kB	169 kB	✓
app-page-exp..prod.js gzip	95.3 kB	95.3 kB	✓
app-page-tur..prod.js gzip	96 kB	96 kB	✓
app-page-tur..prod.js gzip	90.6 kB	90.6 kB	✓
app-page.run...dev.js gzip	142 kB	142 kB	✓
app-page.run..prod.js gzip	89.9 kB	89.9 kB	✓
app-route-ex...dev.js gzip	24.1 kB	24.1 kB	✓
app-route-ex..prod.js gzip	16.7 kB	16.7 kB	✓
app-route-tu..prod.js gzip	16.7 kB	16.7 kB	✓
app-route-tu..prod.js gzip	16.3 kB	16.3 kB	✓
app-route.ru...dev.js gzip	23.5 kB	23.5 kB	✓
app-route.ru..prod.js gzip	16.3 kB	16.3 kB	✓
pages-api-tu..prod.js gzip	9.38 kB	9.38 kB	✓
pages-api.ru...dev.js gzip	9.65 kB	9.65 kB	✓
pages-api.ru..prod.js gzip	9.37 kB	9.37 kB	✓
pages-turbo...prod.js gzip	21.9 kB	21.9 kB	✓
pages.runtim...dev.js gzip	22.5 kB	22.5 kB	✓
pages.runtim..prod.js gzip	21.9 kB	21.9 kB	✓
server.runti..prod.js gzip	49.5 kB	49.5 kB	✓
Overall change	940 kB	940 kB	✓

Diff details

Diff for page.js

Diff too large to display

Diff for edge-ssr.js

Diff too large to display

Commit: 0ba0e76

[mknichel]

cc @huozhi @feedthejim

[feedthejim]

lgtm

[huozhi]

Can we create another test file, override the experimental flags with webpackBuildWorker and parallel options to test? so they can cover both default build and experimental build.
app-dir/app has quite a few test cases, this will loose the default build coverage

[huozhi]

Rename to parallelServerCompiles? "nodejs" and "edge" are runtime but are both "server" target

[mknichel]

Done.

[delbaoliveira]

Formatting:

Suggested change

	The `experimental.parallelServerAndEdgeCompiles` and `experimental.parallelServerBuildTraces`
	options require that the `experimental.webpackBuildWorker` option is set to true. These
	options use workers to improve the paralellization of the build which may improve performance,
	but the build may use more memory at the same time.
	The `experimental.parallelServerAndEdgeCompiles` and `experimental.parallelServerBuildTraces` options require that the `experimental.webpackBuildWorker` option is set to `true`. These options use workers to improve the parallelization of the build, which may improve performance, but the build may use more memory at the same time.

[mknichel]

Thanks for the review @huozhi ! I'm heading OOO and wasn't able to address the test comments yet, but will when I'm back.

[mknichel]

Done.

changes