Extend existing content plugins (CMS integration, middleware, doc gen...)

[mark-tate]

💥 Proposal

I've written a couple of blog posts (which I will shortly publish) on how to use Docusaurus as a headless CMS, with Wordpress's Gutenberg.

I've also written a sample plugin, wordpress-to-docusaurus-plugin which uses GraphQL to pull Blog posts from Wordpress, which I then use to build Docusaurus posts with.

All looks good but one problem remains. Plugins currently load in parallel, so new blog posts are not seen until I reload. I need my plugin's loadContent to run before the blog plugin runs, so it sees the new posts, I've created.

I would like to add an optional loadingPriority to plugin options, which defines the order they are loaded.

This will enable us to still load plugins in parallel or without any priority defined but for certain use-cases, we can prioritise the order of loading.

I've found, I only need to prioritise, loadContent, so we could have a single numeric loadingPriority option which is available to any plugin.

    [
      require.resolve('wordpress-to-docusaurus-plugin'),
      {
        // ... my other options
        loadingPriority: 1 // optional
      },
    ],

It's small change that is required to the existing code to make this possible.
Is this a PR you would be interested in reviewing or do you have an alternative preference ?

Have you read the Contributing Guidelines on issues?

Yes

[slorber]

Hi Mark,

how to use Docusaurus as a headless CMS

You probably mean using Docusaurus as the UI of a headless CMS :)

It's nice to see work being done integrating Docusaurus with WordPress or any other CMS, afaik it's not something many have done so far.

---

What I understand is that your plugin downloads the files as MDX, writes then to the disc, and they are expected to be found by the official blog plugin. So basically the blog plugin (or all others with undefined priority) would somehow need to wait for the completion of your plugin's loadContent, somehow creating "content loading stages".

I don't think it's a good idea to introduce this plugin ordering logic. We really want plugins to work in isolation and not interfere with each other. This ordering logic is also something that was often requested on Gatsby and they pushed hard against implementing it.

---

My suggestion to this problem is that you should not actually write a new Docusaurus plugin for this, but instead enhance the existing blog plugin.

If we created a blog plugin async beforeLoadContent() option, you would be able to write the MDX files here instead of trying to orchestrate a more complex workflow using multiple plugins. Do you think it would work for your usecase?

In an ideal world, I'd like to find a better solution, so that you could use the blog plugin without even having to write .mdx files to the file-system, but we are not there yet.

[mark-tate]

Hi Sebastien

Good to speak again..
Yes, that would work, although, it would only work for blog posts.
You might have folks who want to use a headless CMS to re-create content docs as well.
(The true value here, is using Gutenberg to improve the contribution process, not the content itself)

I took the priority approach to avoid creating too big a change, but what you suggest would also work.
However, we are then pushing responsibility to each plugin, blog, content etc, unless we make it a common API across both.
Given that, there are still some unknowns, that's probably not the easiest place to start ?

I did wonder whether we would want to change the lifecycle, which is a more fundamental change?
In theory, you could also extend the current lifecycle to include before (or even after) lifecycles.

beforeLoadContent:
loadContent:
afterLoadContent:

I could then just create my docs duringbeforeLoadContent and we do not need to change both content and blog plugins.
We could then keep this code, decoupled until we have greater adoption and understand the requirements better.

This is just a simple example of how the current plugin works
https://github.com/mark-tate/wordpress-to-docusaurus-plugin#readme

Might be useful to see the blog, I've written for our spike.
I've not registered a permanent domain for it yet, but pushed it here, so you can see where I am at.

Part 1, is the setup of Wordpress with Docusaurus
Part 2 is how to use your React component in Wordpress and then re-create them in Docusaurus

My thoughts, were to start simple and re-generate the MDX on build.
I was then going to get the build to trigger when a doc updates.

Creating React components in Gutenberg and then exporting them to Docusaurus, works quite well as a demo, our next steps are to see how well this works for a real use-case.

Keen to hear your thoughts.

[slorber]

We could add this to docs and pages as well.

My goal here is to find a good enough workaround to unlock your use case without introducing a new public API that we should not need in the first place.

If we can avoid a new core priority API or core lifecycle APIs I think it's preferable, and I'd rather push this feature to our 3 official content plugins, and find a solution for content plugins to be somehow extensible. And each plugin should probably decide by itself how it can be extended.

This is probably not the most straightforward code, but I think the following could be a pattern we could push forward and support/document better in the future, as it basically does not increase any API surface but let you solve this problem.

const blogPluginExports = require('@docusaurus/plugin-content-blog');

const blogPlugin = blogPluginExports.default;

function blogPluginEnhanced(...pluginArgs) {
  const blogPluginInstance = blogPlugin(...pluginArgs);

  return {
    ...blogPluginInstance,
    loadContent: async function (...loadContentArgs) {

      console.log('will load some WordPress files');
      await new Promise((resolve) => setTimeout(resolve, 1000));
      console.log('WordPress files loaded');

      return blogPluginInstance.loadContent(...loadContentArgs);
    },
  };
}

module.exports = {
  ...blogPluginExports,
  default: blogPluginEnhanced,
};

Somehow you "intercept" the existing plugin lifecycle methods and wrap them with custom logic. I tested it and it works fine.

Maybe the most generic feature we could implement to support this is a "plugin middleware" feature? That would permit to somehow reduce the boilerplate necessary to write the code above to something more simple like:

[
      '@docusaurus/plugin-content-blog',
      {
        ... other options
        middleware: plugin => ({
          ...plugin,
          loadContent: async function (...loadContentArgs) {
     
            console.log('will load some WordPress files');
            await new Promise((resolve) => setTimeout(resolve, 1000));
            console.log('WordPress files loaded');
      
            return plugin.loadContent(...loadContentArgs);
          },
        }),
      },
    ],

We could also create an official helper function to actually provide this plugin lifecycle wrapping feature

Does it make sense?

[mark-tate]

A nice solution, that will work.

I don't think we need any middleware at this stage of things.
We can also follow this approach for presets as well.
I'll update my blog post and complete the spike.

If you come across anyone attempting to do anything similar (or anyone reads this post and wants to contribute) please let me know.

In the meantime, keep you posted on our progress.

Thanks for your help.

[slorber]

Great, let me know if this works for you.

I'll rename this issue, as we agree it's preferable to extend a plugin that to implement a priority api.

We'll see how much people need this and try to get some feedback on an RFC before implementing it.

We can also follow this approach for presets as well.

I'm not 100% sure it will work easily for the preset, but you can disable the preset content plugins with {docs: false, blog: false, pages: false} and add the enhanced plugins manually, or compose the plugins/themes directly.

[RDIL]

I'm going to rebuild the plugin loader to allow plugins to interact with each other.

[slorber]

I'm going to rebuild the plugin loader to allow plugins to interact with each other.

@RDIL not sure what you mean but I don't think it's a good idea, or at least we should talk about this :)

---

Related issue with a good amount of infos on how to extend a content plugin: #4492

[jsamr]

@slorber I am trying to extend the content-docs plugin for API doc generation, and my use-case involves setting a new name for the plugin in order to keep the ability to use the original content-docs plugin for other sections of the site. Unfortunately, the plugin cache path is hardcoded here

docusaurus/packages/docusaurus-plugin-content-docs/src/index.ts

Line 66 in 1ab8aa0

'docusaurus-plugin-content-docs',

And I'm stuck with ENOENT errors... Perhaps the plugin function could take optional arguments for customization.

[slorber]

@jsamr it's hard for me to understand your usecase, you'd rather open another issue with all the details and a simplified example of what you are trying to achieve, that I could run locally

Unfortunately, the plugin cache path is hardcoded here

Maybe using a different pluginId could solve the problem?

Perhaps the plugin function could take optional arguments for customization.

Sure we want to do that but this requires a proper API design

[slorber]

An example integration of the Kentico Kontent CMS with Docusaurus, as a plugin CLI extension (yarn docusaurus sync-kontent):

• https://github.com/kentico-michaelb/kontent-docusaurus-boilerplate
• https://dev.to/kentico-kontent/can-you-evolve-your-documentation-from-fossil-to-fantastic-2fa5

[Josh-Cena]

@kgajera Are you using TypeScript? If so, you don't need the default. You are already importing the default export.

[kgajera]

@kgajera Are you using TypeScript? If so, you don't need the default. You are already importing the default export.

Yes, it is a TypeScript file. Do you mean like this:

[Josh-Cena]

Ah yes, it's because we have a custom module declaration that only exports a bunch of types and doesn't declare the actual plugin function. You need to write a declaration yourself, using our implementation as a reference

[slorber]

Note we don't support yet TS config files / plugin files. If you want to write a plugin in TS, you will have to compile it to JS first before being able to run it.

[niklasp]

Whats the state on this? It would be a cool feature. I just copy and pasted and changed alot in order to create my own plugin with extended pluginData.

[austinbiggs]

Whats the state on this? It would be a cool feature. I just copy and pasted and changed alot in order to create my own plugin with extended pluginData.

I came here to say the same thing, I'm currently working on syncing content from a headless CMS.

[slorber]

My top priority atm is to upgrade some infra (mdx2, React 18). Once it's done I'll work on a few issues including this one to keep increasing the flexibility of Docusaurus

[niklasp]

I jumped on this issue and just want to add my use case here in order to give you a concrete scenario on what a possible upgrade would be cool to cover:

My use case is that I want a landing Page that should display all doc Tags alongside clickable card components that link to 2 docs that have that Tag. Those cards should contain content from the docs, e.g. title, description, a link of course.

The only way I was possible to solve this is to make a copy of the docs plugin and add the functionality there, which was a good learning excercise, and gives me options to change a lot more.

Thanks for looking into this

[liviaerxin]

I am looking into this issue and would like to add my use case to my site:

I want to filter my blogs with multiple tags supported(OR ), adding a new page using dynamic params like http://localhost:3000/blog/tags/?$filter=tag eq 'aa' or tag eq 'bb' .

Thanks for making Docusaurus more accessible and easy to add customized features.

[slorber]

Some useful comments to consider:

• In the future, if you want to "query" plugin data from anywhere, we'll use React Server Components for that (track Support React Server Components #9089). This means that in practice you'll be able to show your latest blog posts on your homepage or even a doc, which is currently not possible due to plugins being sandboxed (they can't see each others data).

• For CMS integration, we'll still need to have an easy way to extend the plugin lifecycles to fetch content from a remote source

[slorber]

Hey 👋

I know it's particularly challenging for the community to understand how to overcome this Docusaurus limitation of plugins not seeing each other's data.

For this reason I created a good example you can take inspiration from, by displaying the recent blog posts on your homepage.

The idea is that you extend the blog plugin to create your own blog plugin. Then you uses the plugin loaded data, and create an additional page (possibly your homepage), injecting into it the props and data you need.

Note: the data you inject as props has to be converted to JSON module bundles first, that's why we call createData() in the plugin. This API is confusing and the DX not great, we'll improve that.

---

Implementation

A runnable example is available here:

https://stackblitz.com/edit/github-crattk?file=custom-blog-plugin.js,docusaurus.config.js,src%2Fcomponents%2FHome%2Findex.js,.gitignore

The implementation has been a bit inspired by this community blog post: https://kgajera.com/blog/display-recent-blog-posts-on-home-page-with-docusaurus/

I hope you'll find it useful.

The implementation relies on these 3 parts:

custom-blog-plugin.js

// ./custom-blog-plugin.js

const blogPluginExports = require('@docusaurus/plugin-content-blog');

const defaultBlogPlugin = blogPluginExports.default;

async function blogPluginExtended(...pluginArgs) {
  const blogPluginInstance = await defaultBlogPlugin(...pluginArgs);

  const pluginOptions = pluginArgs[1];

  return {
    // Add all properties of the default blog plugin so existing functionality is preserved
    ...blogPluginInstance,
    /**
     * Override the default `contentLoaded` hook to access blog posts data
     */
    contentLoaded: async function (params) {
      const { content, actions } = params;

      // Get the 5 latest blog posts
      const recentPostsLimit = 5;
      const recentPosts = [...content.blogPosts].splice(0, recentPostsLimit);

      async function createRecentPostModule(blogPost, index) {
        console.log({ blogPost });
        return {
          // Inject the metadata you need for each recent blog post
          metadata: await actions.createData(
            `home-page-recent-post-metadata-${index}.json`,
            JSON.stringify({
              title: blogPost.metadata.title,
              description: blogPost.metadata.description,
              frontMatter: blogPost.metadata.frontMatter,
            })
          ),

          // Inject the MDX excerpt as a JSX component prop
          // (what's above the <!-- truncate --> marker)
          Preview: {
            __import: true,
            // The markdown file for the blog post will be loaded by webpack
            path: blogPost.metadata.source,
            query: {
              truncated: true,
            },
          },
        };
      }

      actions.addRoute({
        // Add route for the home page
        path: '/',
        exact: true,

        // The component to use for the "Home" page route
        component: '@site/src/components/Home/index.js',

        // These are the props that will be passed to our "Home" page component
        modules: {
          homePageBlogMetadata: await actions.createData(
            'home-page-blog-metadata.json',
            JSON.stringify({
              blogTitle: pluginOptions.blogTitle,
              blogDescription: pluginOptions.blogDescription,
              totalPosts: content.blogPosts.length,
              totalRecentPosts: recentPosts.length,
            })
          ),
          recentPosts: await Promise.all(
            recentPosts.map(createRecentPostModule)
          ),
        },
      });

      // Call the default overridden `contentLoaded` implementation
      return blogPluginInstance.contentLoaded(params);
    },
  };
}

module.exports = {
  ...blogPluginExports,
  default: blogPluginExtended,
};

docusaurus.config.js

{
// ...
plugins: [
    // Use custom blog plugin
    [
      './custom-blog-plugin',
      {
        id: 'blog',
        routeBasePath: 'blog',
        path: './blog',
        blogTitle: 'My Awesome Blog',
        blogDescription: 'A great blog with homepage Docusaurus integration',
      },
    ],
  ],

  presets: [
    [
      'classic',
      /** @type {import('@docusaurus/preset-classic').Options} */
      ({
        docs: {
          sidebarPath: require.resolve('./sidebars.js'),
          // Please change this to your repo.
          // Remove this to remove the "edit this page" links.
          editUrl:
            'https://github.com/facebook/docusaurus/tree/main/packages/create-docusaurus/templates/shared/',
        },
        blog: false,
        theme: {
          customCss: require.resolve('./src/css/custom.css'),
        },
      }),
    ],
  ],
// ...

}

src/components/Home/index.js

import React from 'react';
import clsx from 'clsx';
import Link from '@docusaurus/Link';
import useDocusaurusContext from '@docusaurus/useDocusaurusContext';
import Layout from '@theme/Layout';
import HomepageFeatures from '@site/src/components/HomepageFeatures';
import styles from './index.module.css';

function HomepageHeader() {
  const { siteConfig } = useDocusaurusContext();
  return (
    <header className={clsx('hero hero--primary', styles.heroBanner)}>
      <div className="container">
        <h1 className="hero__title">{siteConfig.title}</h1>
        <p className="hero__subtitle">{siteConfig.tagline}</p>
        <div>
          <Link
            className="button button--secondary button--lg"
            to="/docs/intro"
          >
            Docusaurus Tutorial - 5min ⏱️
          </Link>
        </div>
      </div>
    </header>
  );
}

function RecentBlogPostCard({ recentPost }) {
  const { Preview, metadata } = recentPost;
  return (
    <article style={{ padding: 20, marginTop: 20, border: 'solid thick red' }}>
      <h2>{metadata.title}</h2>
      <p>{metadata.description}</p>
      <p>FrontMatter: {JSON.stringify(metadata.frontMatter)}</p>
      <hr />
      <Preview />
    </article>
  );
}

export default function Home({ homePageBlogMetadata, recentPosts }) {
  const { siteConfig } = useDocusaurusContext();
  console.log({ homePageBlogMetadata, recentPosts });
  return (
    <Layout
      title={`Hello from ${siteConfig.title}`}
      description="Description will go into a meta tag in <head />"
    >
      <HomepageHeader />
      <main style={{ padding: 30 }}>
        <h1>{homePageBlogMetadata.blogTitle}</h1>
        <p>{homePageBlogMetadata.blogDescription}</p>
        <p>
          Displaying some sample posts:
          {homePageBlogMetadata.totalRecentPosts} /{' '}
          {homePageBlogMetadata.totalPosts}
        </p>

        <section>
          {recentPosts.map((recentPost, index) => (
            <RecentBlogPostCard key={index} recentPost={recentPost} />
          ))}
        </section>
        <hr />
        <HomepageFeatures />
      </main>
    </Layout>
  );
}

[samducker]

Hi @slorber

apologies if this has already been answered elsewhere I could not find the answer and just found this thread from a couple of years ago.

I am currently working on a documentation site for a client, who will also have non-technical users editing the product/platform side of the documentation.

I am currently using spinalcms.com (a file based / git cms similar to netlify cms) however it has limitations on using MDX and markdown tables which is not ideal as I want the documentation to be interactive and amazing in all ways :)

Anyways, I've used headless cms like storyblok/contentful/sanity before etc which can be really flexible to whatever you want to do with functions in next.js like getStaticPaths and getStaticProps.

I wondered if there was any solution to use a headless cms for docs content like this or advice you could give.

My only core idea so far was I could create a pre-build script in my package.json which would run before the docusarus build and generate the markdown files from the headless cms api.

I don't know if this is a better idea than creating a plugin.

Thanks in advance.

p.s. my use case is more related to documentation pages as opposed to rendering a blog or custom page.

[slorber]

My only core idea so far was I could create a pre-build script in my package.json which would run before the docusarus build and generate the markdown files from the headless cms api.

@samducker that's what most CMS + Docusaurus users do today: as long as you get the markdown files on the filesystem it should work.

We'll explore how to provide a way to integrate with CMS without this intermediate build step in the future but for now it's only only way that does not involve retro-engineering our plugin's code.

[colececil]

Maybe the most generic feature we could implement to support this is a "plugin middleware" feature?

I really like this middleware idea proposed by @slorber above. This would also fix the "infinite loop on refresh" issue for custom plugins that provide data to put in Docs. I managed to find a workaround for the issue (see this StackOverflow answer I just wrote), but this proposed feature would make the problem much easier to solve.

My personal use case is not with a CMS as a source, but another subproject in the same monorepo. The monorepo has three subprojects - a JS API library, a static site for testing and demoing the library, and a Docusaurus site with API documentation. I wanted to reuse the "demo site" code as "example API usage code" in the Docusaurus site, so I wrote a plugin to copy the source code from the other subproject and write it to the docs directory of the Docusaurus project as Markdown.