v2: Allow custom extensions to be passed into the Webpack file loader

[mattisdada]

🚀 Feature

The ability to provide your own list of file extensions to be loaded via file loader or a specific directory that will always be loaded in via the file loader.

Have you read the Contributing Guidelines on issues?

Yes

Motivation

I currently have a YAML file that needs to be downloaded via the static file server, currently the only way to get download paths are by import or require, but the file loader is limited to pdf|doc|docx|xls|xlsx|zip|rar (reference) currently with no way of adding to that list. This will cause the Webpack loader to panic with a You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. error.

Handling files that are not in this very limited list are impossible to use with Docusaurus for generating download links, this limits the functionality greatly.

Example

import InsomniaUrl from "@site/static/downloads/insomnia.yml";
// You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file.

Pitch

There should be a configuration option in docusaurus.config.js to allow setting a custom list (or appending to the existing list) of allowed file types to be used by the file loader. Alternatively a path (ie static/downloads) where all files get handled by the file loader, this path should be configurable.

[mattisdada]

My workaround for this issue was to do the following:

import useBaseUrl from "@docusaurus/useBaseUrl";

export const InsomniaButton = () => {
  const fileUrl = "/downloads/insomnia.yml";
  const url = `${location.protocol}//${location.host}${useBaseUrl(fileUrl)}`;
  const encodedUrl = encodeURIComponent(url);
// ...

There might be some other Docusaurus hook that allows generating the full url without relying on location, but I wasn't able to find it...

But this approach is just string juggling which means we don't get the safety of using the file loader and requires using BrowserOnly as a workaround

[slorber]

Handling files that are not in this very limited list are impossible to use with Docusaurus for generating download links, this limits the functionality greatly.

That's not entirely true, you can add whatever you want to the webpack config with configurreWebpack, including file-loader configs, so if we add this, it would only be a "shortcut" for something that is already possible today.

Something like this might work:

module.exports = {
  configureWebpack: function () {
    return {
      module: {
        rules: [
          {
            test: /\.yaml$/,
            use: [
              {
                loader: 'file-loader',
                options: {name: 'assets/files/[name]-[hash].[ext]'},
              },
            ],
          },
        ],
      },
    };
  },
};

Note we have multiple output folders for static assets currently. If we enabled to configure a list of extensions, do we want all these files to be in build/files?

[mattisdada]

@slorber Thanks for that, I looked around and I couldn't find the Webpack escape hatch in the docs 😅.

I think it's still worth while for something that is a bit more systematic for this, maybe I overestimate how often docs are generated with downloaded files?

Otherwise I'm perfectly happy for this issue to be just closed, your answer is a good solution.

[slorber]

Thanks for the feedback

I think it's still worth while for something that is a bit more systematic for this, maybe I overestimate how often docs are generated with downloaded files?

This is the first time anyone request that, so I'll just close for now, and see if this is requested again in the future

[khusamov]

@slorber

Hello! I tried to place configureWebpack in the docusaurus.config file.js
And got an error:

[ERROR] Error: These field(s) ("configureWebpack",) are not recognized in docusaurus.config.js.
If you still want these fields to be in your configuration, put them in the "customFields" field.

Where should I put the configureWebpack?

[Josh-Cena]

@khusamov configureWebpack is a plugin lifecycle. See:

• https://docusaurus.io/docs/advanced/plugins#function-definition
• https://docusaurus.io/docs/api/plugin-methods/lifecycle-apis#configureWebpack

If you want to modify docusaurus.config.js instead of creating a separate plugin file, you would write something like:

module.exports = {
  plugins: [
    () => ({
      configureWebpack() {
        return {...};
      },
    }),
  ],
};

[benjie]

I needed to import a file from the filesystem into the documentation to be included raw, to do so I added the following Docusaurus plugin to the list of plugins in docusaurus.config.js:

const webpackAssetSourcePlugin = () => ({
  configureWebpack() {
    return {
      module: {
        rules: [
          {
            resourceQuery: /raw/,
            type: "asset/source",
          },
        ],
      },
    };
  },
}),

The plugin uses Webpack 5's asset/source type for any imports with ?raw at the end. Then in my .mdx I could import the content by being sure to append ?raw:

import Mermaid from '@theme/Mermaid';
import chart from '../path/to/file.mermaid?raw';

# Blah

Blah blah blah

<Mermaid chart={chart}/>

Thanks for all the guidance above ❤️

[slorber]

great to know that you made this work

I tried to migrate Docusaurus from file-loader to the assets system but there were some annoying edge cases that prevented from completing this migration (#4708)

also discussed here: #8106

[erzz]

@slorber (and anyone else interested!) - I would love something to be easier to configure here. Frankly as a relative n00b coming to webpack5 I barely understand what you guys say in terms of solutions here :)

I come from Hugo where the idea of a data/ directory is just automatically accessible with a simple inline command like {{ $data := (index somefile.yml) }}

Literally all I want to do is load content from .yml files in the site (its actually a collection of gihub workflows) and render various parts.

Even a more laborious method (yaml-js + fs) isn't working because also webpack doesnt seem to like the fs package

There are many I am sure that have a local datasource that is a bunch of YAML or JSON files and to be honest its a whole world of pain for noobs like me to get something working

The only little success I have had is with something like:

import wf from "yaml-loader!@site/.github/workflows/artillery.yml";

export default function JobList(props) {
  return (
    <div>
      <h3>Workflow</h3>
      <pre>
        {JSON.stringify(wf.jobs, null, 2)}
      </pre>
    </div>
  );
}

But that is difficult to work with in a component way as I need to pass it in from props

But to return to the issue - surely command data exchange formats such as YAML should be in the OOTB list?

[slorber]

@erzz there's not a single way to import a yaml file:

• file-loader (what other users wanted): the imported yaml becomes a public URL string
• yaml-loader (what you want?): the imported yaml becomes a json object

We can't satisfy both users at the same time, and in any case you can configure this yourself with the configureWebpack hook

I'm open to supporting Yaml OOTB better. But if we supported Yaml out of the box, but not in the way you though we should (because there's not a single use-case), it could even make it harder for you to revert our OOTB yaml support and add your own support, and vice versa. I hope you understand my concerns about doing so and that enabling you to bring your own support is the safest option for us, even if it's less convenient for a few users.

[erzz]

Thanks @slorber - totally understand that it becomes complex in keeping everyone happy with OOTB

So can we we instead have some n00b documentation for webpack configuration (and I get that its likely described very well @ webpack :) ) - but in the context of docusaurus and the default configuration (where all config is in const config and thus looking nothing like the 99% of examples floating around)?

[slorber]

@erzz I don't want to actually document any Webpack thing because it's not sure we'll stick with Webpack forever.

If you want to use them, you should feel the pain and understand that you are using something that is clearly "unofficial", and may disappear later in any upcoming Docusaurus release. If we document it, we encourage you to use things that will go away some day.

By not documenting it on purpose, we actually reduce the long-term overall pain of our whole user base.

[erzz]

Haha! Fair enough!

[BenjaminYde]

Is it possible to configure it for .pdf files?
I can't find a solution for that.

[slorber]

Afaik we already support file loader for PDF files.

    otherAssets: () => ({
      use: [loaders.file({folder: 'files'})],
      test: /\.(?:pdf|docx?|xlsx?|zip|rar)$/i,
    }),

If you think we don't, please provide a repro.

[BenjaminYde]

The following is my problem, I just want to use docusaurus v3.1.1 and a pdf viewer.
Hope you can help me any further / see anything that doesn't seem right.

Log after npm run build:

Error: Unable to build website for locale en.
    at tryToBuildLocale (/workspace/docusaurus/node_modules/@docusaurus/core/lib/commands/build.js:55:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async mapAsyncSequential (/workspace/docusaurus/node_modules/@docusaurus/utils/lib/jsUtils.js:44:24)
    at async Command.build (/workspace/docusaurus/node_modules/@docusaurus/core/lib/commands/build.js:82:21) {
  [cause]: Error: Failed to compile due to Webpack errors.
  Module parse failed: Unexpected character '' (1:0)
  You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
  (Source code omitted for this binary file)
      at /workspace/docusaurus/node_modules/@docusaurus/core/lib/webpack/utils.js:207:24
      at /workspace/docusaurus/node_modules/webpack/lib/MultiCompiler.js:554:14
      at processQueueWorker (/workspace/docusaurus/node_modules/webpack/lib/MultiCompiler.js:491:6)
      at process.processTicksAndRejections (node:internal/process/task_queues:77:11)

pdfviewer.js:

import React, { useState } from 'react';
import { Document, Page, pdfjs } from 'react-pdf';
import 'react-pdf/dist/esm/Page/AnnotationLayer.css';

// Set worker src.
pdfjs.GlobalWorkerOptions.workerSrc = `https://cdnjs.cloudflare.com/ajax/libs/pdf.js/${pdfjs.version}/pdf.worker.js`;

const PdfViewer = ({ file, width, fromPage, toPage }) => {
  const [numPages, setNumPages] = useState(null);

  function onDocumentLoadSuccess({ numPages: loadedNumPages }) {
    setNumPages(loadedNumPages);
  }

  return (
    <div>
      <Document
        file={file}
        onLoadSuccess={onDocumentLoadSuccess}
      >
        {numPages && Array.from(
          new Array(numPages),
          (_, index) => index + 1
        )
        .filter(pageNumber => (!fromPage || pageNumber >= fromPage) && (!toPage || pageNumber <= toPage)) 
        .map(pageNumber => (
          <Page
            key={`page_${pageNumber}`}
            pageNumber={pageNumber}
            width={width}
            renderTextLayer={false} 
          />
        ))}
      </Document>
    </div>
  );
};

export default PdfViewer;

myfile.mdx:

import PdfViewer from '@site/src/components/pdfviewer';

<PdfViewer
  file={require("/files/my_file.pdf")}
  width={1000}
  fromPage={2}
  toPage={6}
/>

In my docusaurus.config.js

  plugins: [
    [
     // Custom plugin to modify Webpack config
    function myCustomPlugin(context, options) {
      return {
        name: 'custom-webpack-plugin',
        configureWebpack(config, isServer, utils, content) {
          return {
            module: {
              rules: [
                {
                  test: /\.pdf$/,
                  type: 'asset/resource',
                },
              ],
            },
          };
        },
      };
    },
  ],

[slorber]

I only help if you create a runnable minimal repro that I can run in a sandbox.

If it is a loader problem, then show me a repro that does not involve react-pdf and prove me that you can't require pdf files.

[BenjaminYde]

I was able to make it work with the following config:

plugins: [
    [
    // Custom plugin to modify Webpack config
    function myCustomPlugin(context, options) {
      return {
        name: 'custom-webpack-plugin',
        configureWebpack(config, isServer, utils, content) {
          return {
            module: {
              rules: [
                {
                  test: /\.pdf$/,
                  use: ["file-loader"],
                },
                {
                  test: /\.node$/,
                  use: ["node-loader"],
                }
              ],
            },
          };
        },
      };
    },
  ],

[slorber]

ok, but I doubt any of this is actually necessary 🤷‍♂️ it should work without any extra plugin/config

[CMCDragonkai]

I want to be able to load gltf, bin, obj, usdz, glb.

So I added this in:

    configureWebpack(config, isServer) {
      return {
        module: {
          rules: [
            {
              test: /\.(gltf|bin|obj|usdz|glb)$/,
              use: [
                {
                  loader: 'file-loader',
                  options: {
                    outputPath: 'assets/files',
                  },
                },
              ],
            },
          ],
        },
      };
    },

But this does some strange stuff, the txt is forgotten about so it now fails on robots.txt. I have to add in the txt in the extensions.

But the result is always:

/public/assets/_/files/

Why is there a _ being produced?

[CMCDragonkai]

Ok I realised I followed an incorrect guide on how to do this, so now I can create my own plugin loader for it.

However what's the generic way of saying any given path being loaded should be preserved under assets?

[CMCDragonkai]

Was the .txt extension ever understood by webpack?