[DO-NOT-MERGE] RFC: use esbuild to significantly speedup building #4487

SamChou19815 · 2021-03-21T16:34:22Z

Motivation

This PR tries to use esbuild to compile all the JS code (including all the mdx-react produced js code). Since esbuild is written in Go with performance in mind, it results in a much faster build. Locally, the server build takes 1min for each locale without this change, and only 40s with this change.

However, I do recognize that this PR is not ready for merge right now. I understand that many websites might depend on using specific babel config, and probably esbuild's min target es2015 is not good enough. The maintainers might also want to put this behind a flag (either in docusaurus.config.js or a env-var). I only created this PR as a proof of concept how it can be done and to show how fast it can speed up build in CI, and wait on maintainers' opinion on whether and how we want to move forward with this. Therefore, I haven't setup any gating yet and simply replaces babel config and minimize plugin with esbuild stuff.

I'm open to keep this behind a flag or only as a plugin that imperatively patch the webpack config, like the one I used for my personal website.

Have you read the Contributing Guidelines on pull requests?

Yes

Test Plan

CI

Related PRs

(If this PR adds or changes functionality, please take some time to update the docs at https://github.com/facebook/docusaurus, and link to your PR here.)

netlify · 2021-03-21T16:37:30Z

[V1] Deploy preview failure

Built without sensitive environment variables with commit c443ac8

https://app.netlify.com/sites/docusaurus-1/deploys/6060e7e54b1f0b000810ae52

netlify · 2021-03-21T16:41:29Z

Deploy preview for docusaurus-2 failed.

Built without sensitive environment variables with commit c443ac8

https://app.netlify.com/sites/docusaurus-2/deploys/6060e7e53b173a0008157cd9

github-actions · 2021-03-21T16:41:50Z

⚡️ Lighthouse report for the changes in this PR:

Category	Score
🟠 Performance	79
🟢 Accessibility	96
🟢 Best practices	100
🟢 SEO	100
🟢 PWA	95

Lighthouse ran on https://deploy-preview-4487--docusaurus-2.netlify.app/

SamChou19815 · 2021-03-21T16:42:25Z

This PR improves the build time of server from 2.90m to 1.92m in this PR, that's a 30% reduction. (See the build size log)

lex111

Yes, yes, we definitely need it! 👍 I thought about esbuild using too, especially when I looked at benchmark results https://github.com/privatenumber/minification-benchmarks#-results
Very impressive, we have to give it a chance. Manage it via env var as an experimental feature is good idea.

armano2 · 2021-03-22T03:32:51Z

This PR improves the build time of server from 2.92m in previous PR to 1.61m in this PR

bdw, both links point to same buld

SamChou19815 · 2021-03-22T03:36:50Z

@armano2 whoops, updated it

armano2 · 2021-03-22T03:39:17Z

Size Change: +3.81 MB (+663%) 🆘

Total Size: 4.39 MB

Filename	Size	Change
`website/build/assets/css/styles.********.css`	87.1 kB	-48 B (0%)
`website/build/assets/js/main.********.js`	4.22 MB	+3.82 MB (+951%)	🆘
`website/build/blog/2017/12/14/introducing-docusaurus/index.html`	60.3 kB	-498 B (-1%)
`website/build/index.html`	25.5 kB	-242 B (-1%)

ℹ️ View Unchanged

Filename	Size	Change
`website/build/docs/introduction/index.html`	235 B	0 B

_{compressed-size-action}

SamChou19815 · 2021-03-22T04:14:29Z

~~Looks like the bloated the size is due to esbuild not removing license comments... Might need other hacks for that...~~

Nevermind, the problem is actually code-splitting is broken. Now it's fixed, and we even have some size reductions:

Filename	Size	Change
`website/build/assets/js/main.********.js`	397 kB	-4.67 kB (-1%)

ℹ️ View Unchanged

Filename	Size	Change
`website/build/assets/css/styles.********.css`	87.2 kB	0 B
`website/build/blog/2017/12/14/introducing-docusaurus/index.html`	60.8 kB	0 B
`website/build/docs/introduction/index.html`	235 B	0 B
`website/build/index.html`	25.7 kB	0 B

Turns out I need this change to overcome this issue in esbuild.

   return {
     loader: require.resolve('esbuild-loader'),
-    options: {loader: 'tsx', format: 'cjs', target: 'es2015'},
+    options: {
+      loader: 'tsx',
+      format: isServer ? 'cjs' : 'esm',
+      target: 'es2017',
+    },
   };
 }

SamChou19815 · 2021-03-22T18:09:51Z

packages/docusaurus/src/webpack/utils.ts

-      },
-    }),
-  ];
+  const minimizer = [new ESBuildMinifyPlugin({target: 'es2017'})];


Switched output to es2017, since v2 doesn't plan to support ie11, and all other major browsers have good support for es2017: https://caniuse.com/?search=es2017

SamChou19815 · 2021-03-22T21:10:58Z

packages/docusaurus-plugin-content-docs/package.json

+    "build": "tsc && yarn build-theme-esm && yarn build-client-esm",
+    "build-theme-esm": "esbuild src/theme/hooks/useDocs.ts > lib/theme/hooks/useDocs.js",
+    "build-client-esm": "esbuild src/client/docsClientUtils.ts > lib/client/docsClientUtils.js",


Somehow esbuild doesn't like that the client files are not in esm, so I setup the script to compile the client files to esm.

SamChou19815 · 2021-03-22T21:27:55Z

packages/docusaurus-theme-live-codeblock/src/custom-buble.js

-const {transform, features: bubleFeatures} = require('@philpl/buble');
+import {transform, features as bubleFeatures} from '@philpl/buble';

-// This file is designed to mimic what's written in
-// https://github.com/kitten/buble/blob/mini/src/index.js, with custom transforms options,
-// so that webpack can consume it correctly.
-exports.features = bubleFeatures;
-
-exports.transform = function customTransform(source, options) {
+function customTransform(source, options) {
  return transform(source, {...options, transforms: {asyncAwait: false}});
-};
+}
+
+export {bubleFeatures as features, customTransform as transform};


Need to turn this into esm to make esbuild happy.

SamChou19815 · 2021-03-23T00:33:56Z

packages/docusaurus/src/webpack/utils.ts

 export function getBabelLoader(
  isServer: boolean,
+  // eslint-disable-next-line @typescript-eslint/no-unused-vars
  babelOptions?: TransformOptions | string,
 ): Loader {
  return {
-    loader: require.resolve('babel-loader'),
-    options: getBabelOptions({isServer, babelOptions}),
+    loader: require.resolve('esbuild-loader'),
+    options: {
+      loader: 'tsx',
+      format: isServer ? 'cjs' : 'esm',
+      target: 'es2017',
+    },
  };
 }


I plan to make this function be like

if (process.env.ESBUILD) { return { loader: require.resolve('esbuild-loader'), options: { loader: 'tsx', format: isServer ? 'cjs' : 'esm', target: 'es2017', }, }; return { loader: require.resolve('babel-loader'), options: getBabelOptions({isServer, babelOptions}), } }

In this way all the existing code that uses the getBabelLoader can work without modification. This is somehow needed since this function is exposed here. However, this is bad since the function should then really be named as getJSLoader. Or we can create a new function called getJSLoader and deprecate the getBabelLoader function.

Do maintainers have any options?

@lex111 @slorber

getJSLoader is also something I had in mind. Would be fine to deprecate getBabelLoader()

SamChou19815 · 2021-03-28T21:45:41Z

looks like the recent bump of esbuild breaks a lot of things...

SamChou19815 · 2021-03-28T22:03:53Z

Closing this now... looks like patching the webpack config doesn't quite work since there are more moving parts in the SSG part that I don't understand. I may revisit this when I figure out how things work.

slorber · 2021-03-29T15:31:09Z

packages/docusaurus/src/webpack/utils.ts

 export function getBabelLoader(
  isServer: boolean,
+  // eslint-disable-next-line @typescript-eslint/no-unused-vars
  babelOptions?: TransformOptions | string,
 ): Loader {
  return {
-    loader: require.resolve('babel-loader'),
-    options: getBabelOptions({isServer, babelOptions}),
+    loader: require.resolve('esbuild-loader'),
+    options: {
+      loader: 'tsx',
+      format: isServer ? 'cjs' : 'esm',
+      target: 'es2017',
+    },
  };
 }


getJSLoader is also something I had in mind. Would be fine to deprecate getBabelLoader()

slorber · 2021-03-29T15:33:10Z

packages/docusaurus/src/webpack/utils.ts

-  }
-
-  return minimizer;
+  return [new ESBuildMinifyPlugin({target: 'es2017', css: true})];


Wonder how much different this minification is compared to the old one?

Earlier stat because I mess up the build shows that it reduces the output by 1%.

slorber · 2021-03-29T15:37:51Z

Thanks @SamChou19815 , that's nice to see that integrating esbuild with our Webpack setup is not too complicated and to have some perf numbers.

On the short term the plan is to merge the Webpack 5 migration (#4089), add auto-generated sidebars and prepare a beta release.

After that, I would be happy to enable an experimental esbuild setup, and maybe we could even make it the default if it is successful and the site does not have any custom babel config (or if it only contains our preset?).

Something important to note: I'd like the i18n <Translate> component to be a "zero-runtime" API and have a Babel plugin inline the translations directly in the page markup instead of loading a global i18n bundle (not correctly code-splitted). We'll have to figure out if this is possible with esbuild, as afaik it may provide fewer capabilities compared to Babel?

SamChou19815 · 2021-03-29T18:01:04Z

Something important to note: I'd like the i18n <Translate> component to be a "zero-runtime" API and have a Babel plugin inline the translations directly in the page markup instead of loading a global i18n bundle (not correctly code-splitted). We'll have to figure out if this is possible with esbuild, as afaik it may provide fewer capabilities compared to Babel?

I think we can add a loader that deals with this before the esbuild loader?

SamChou19815 · 2021-03-29T18:02:23Z

Redoing it in #4532.

slorber · 2021-03-29T18:21:56Z

I think we can add a loader that deals with this before the esbuild loader?

Yeah, we could try, but that would somehow defeat the purpose of using esbuild in the first place as it requires parsing twice the JS code? Not sure how expensive it would be

SamChou19815 · 2021-03-29T18:47:13Z

Yeah, we could try, but that would somehow defeat the purpose of using esbuild in the first place as it requires parsing twice the JS code? Not sure how expensive it would be

I think it's truly unavoidable to run JS code in this case, unless we also write the same transformation code in go as a esbuild go plugin, which I think it's too much to maintain. We can potentially avoid double parsing by detecting whether there is import on @docusaurus/Translate or <Translate...

slorber · 2021-03-30T08:34:05Z

We don't have to implement this optim soon anyway, it's not a big deal for now, but would just like to keep the opportunity to solve this 18n code-splitting problem someday :) If esbuild integration is successful we could build a go plugin in 6 months or 1 year to optimize things for Docusaurus even more.

SamChou19815 requested review from lex111 and slorber as code owners March 21, 2021 16:34

SamChou19815 marked this pull request as draft March 21, 2021 16:34

facebook-github-bot added the CLA Signed Signed Facebook CLA label Mar 21, 2021

lex111 reviewed Mar 21, 2021

View reviewed changes

SamChou19815 commented Mar 22, 2021

View reviewed changes

SamChou19815 commented Mar 23, 2021

View reviewed changes

SamChou19815 marked this pull request as ready for review March 23, 2021 00:34

RFC: use esbuild to significantly speedup building

c443ac8

SamChou19815 closed this Mar 28, 2021

SamChou19815 deleted the esbuild branch March 28, 2021 22:04

slorber reviewed Mar 29, 2021

View reviewed changes

SamChou19815 mentioned this pull request Mar 29, 2021

[DO-NOT-MERGE] RFC: use esbuild to significantly speedup building #4532

Closed

SamChou19815 mentioned this pull request May 13, 2021

feat(v2): Allow customization of js loader, replace babel by esbuild in Docusaurus website #4766

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DO-NOT-MERGE] RFC: use esbuild to significantly speedup building #4487

[DO-NOT-MERGE] RFC: use esbuild to significantly speedup building #4487

SamChou19815 commented Mar 21, 2021

netlify bot commented Mar 21, 2021 •

edited

Loading

netlify bot commented Mar 21, 2021 •

edited

Loading

github-actions bot commented Mar 21, 2021 •

edited

Loading

SamChou19815 commented Mar 21, 2021 •

edited

Loading

lex111 left a comment

armano2 commented Mar 22, 2021

SamChou19815 commented Mar 22, 2021

armano2 commented Mar 22, 2021

SamChou19815 commented Mar 22, 2021 •

edited

Loading

SamChou19815 Mar 22, 2021

SamChou19815 Mar 22, 2021 •

edited

Loading

SamChou19815 Mar 22, 2021

SamChou19815 Mar 23, 2021

SamChou19815 Mar 24, 2021

slorber Mar 29, 2021

SamChou19815 commented Mar 28, 2021

SamChou19815 commented Mar 28, 2021

slorber Mar 29, 2021

slorber Mar 29, 2021

SamChou19815 Mar 29, 2021

slorber commented Mar 29, 2021

SamChou19815 commented Mar 29, 2021

SamChou19815 commented Mar 29, 2021

slorber commented Mar 29, 2021

SamChou19815 commented Mar 29, 2021

slorber commented Mar 30, 2021 •

edited

Loading

[DO-NOT-MERGE] RFC: use esbuild to significantly speedup building #4487

[DO-NOT-MERGE] RFC: use esbuild to significantly speedup building #4487

Conversation

SamChou19815 commented Mar 21, 2021

Motivation

Have you read the Contributing Guidelines on pull requests?

Test Plan

Related PRs

netlify bot commented Mar 21, 2021 • edited Loading

netlify bot commented Mar 21, 2021 • edited Loading

github-actions bot commented Mar 21, 2021 • edited Loading

SamChou19815 commented Mar 21, 2021 • edited Loading

lex111 left a comment

Choose a reason for hiding this comment

armano2 commented Mar 22, 2021

SamChou19815 commented Mar 22, 2021

armano2 commented Mar 22, 2021

SamChou19815 commented Mar 22, 2021 • edited Loading

Choose a reason for hiding this comment

SamChou19815 Mar 22, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SamChou19815 commented Mar 28, 2021

SamChou19815 commented Mar 28, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slorber commented Mar 29, 2021

SamChou19815 commented Mar 29, 2021

SamChou19815 commented Mar 29, 2021

slorber commented Mar 29, 2021

SamChou19815 commented Mar 29, 2021

slorber commented Mar 30, 2021 • edited Loading

netlify bot commented Mar 21, 2021 •

edited

Loading

netlify bot commented Mar 21, 2021 •

edited

Loading

github-actions bot commented Mar 21, 2021 •

edited

Loading

SamChou19815 commented Mar 21, 2021 •

edited

Loading

SamChou19815 commented Mar 22, 2021 •

edited

Loading

SamChou19815 Mar 22, 2021 •

edited

Loading

slorber commented Mar 30, 2021 •

edited

Loading