KaTeX expressions with escapes and braces throw errors

[hugmanrique]

Subject of the issue

I'm trying to setup a Next.js site with mdx and these remark plugins:

• [email protected]
• @kwangkim/[email protected] (I couldn't find the module's source on GitHub, but you can view it on this RunKit)

And this file under pages/:

This is some extra text that gets rendered correctly if I remove the KaTeX block below:

$$
23_{10} = 2^4+2^2+2^1+2^0 = \underbrace{10111_2}_{\mathrm{binary\ representation}}
$$

I assume the escapes inside braces ({}) are confusing MDX and it's trying to parse them as a expression, which causes the following error:

(gist error)

As you can see, it's complaining about a "bad character escape sequence". One way to fix this would be to ignore all the braces and escapes {}\ inside $ (inlineMath) or $$ (math) blocks.

Your environment

• Windows 10 Pro 1803
• @zeit/next-mdx: 1.2.0
  • @mdx-js/mdx: 0.15.7
• next: 7.0.2-canary.49

Steps to reproduce

Create new Next.js project with the following next.config.js file:

const remarkMath = require('remark-math');
const katex = require('@kwangkim/remark-jsx-katex');

module.exports = require('@zeit/next-mdx')({
  options: {
    mdPlugins: [remarkMath, katex],
  },
  pageExtensions: ['js', 'mdx']
});

Now, create pages/foo.mdx:

This is some extra text that gets rendered correctly if I remove the KaTeX block below:

$$
23_{10} = 2^4+2^2+2^1+2^0 = \underbrace{10111_2}_{\mathrm{binary\ representation}}
$$

Now, run next and load localhost:3000/foo in your browser.

Expected behaviour

The KaTeX escapes should be ignored even if they are inside braces ({}).

Actual behaviour

The KaTeX escapes and generally, all the content inside braces doesn't get ignored, causing compiling and React rendering issues.

[ChristopherBiscardi]

We ran into this in gatsby-mdx too (actually we sort of run into things like this a lot because of the way the gatsby plugins function). I added some replacement code in a babel plugin that we use to post-process MDX content after remark and hast plugins have run but before we ship the JSX.

if (
  path.node.openingElement.name.name ===
  "annotation" /* && if gatsby-remark-katex is enabled */
) {
  const genCode = path.node.children.map(ast => generate.default(ast).code);
  path.node.children = [
    t.jSXText(
      genCode
        .join("")
        .replace("{", "{")
        .replace("}", "}")
    )
  ];
}

As you can see, it's complaining about a "bad character escape sequence". One way to fix this would be to ignore all the braces and escapes {}\ inside $ (inlineMath) or $$ (math) blocks.

There's a similar "javascript string escaping"-related issue in #366. so I wonder if we should keep adding to the list of "non-escaped sections" (code blocks, $ or $$ sequences, etc) or somehow default to "html style" interpretation for special chars like {, \u, etc and force \{ or something for bracket literals to get passed through into the final jsx? dunno, haven't had a moment to think through the ramifications here all the way.

[hugmanrique]

I tried copying that file into a babelPlugin.js local file, changed Next.js .babelrc file to:

{
  "presets": [
    "next/babel"
  ],
  "plugins": [
    "./build/babelPluginHtmlAttrToJsxAttr.js"
  ]
}

And it didn't work. I tried adding debugging messages and it turns out the JSXElement visitor is not getting called, might be related to plugin execution order, but I'm not a Babel expert so I don't know how to fix it and if this is even the error cause :/

On another note, I think I found a bug in your babel plugin implementation:

The replace calls with a string will only replace the first occurance (MDN docs), so you will need to replace it to a regex expression and pass the g flag to the two replace calls.

[ChristopherBiscardi]

And it didn't work.

yeah, probably because in gatsby-mdx we apply it to the MDX output before returning it for the user's babel config to run on. It's used as some internal infrastructure, not as a plugin inline with whatever's in next/babel (for example). The equivalent place for next looks like it would be between the mdx loader and the babel loader in next/babel, but it also looks like that's not configurable to the user.

If I had to guess, I'd assume that the JSX expressions are already compiled to function calls by something in the preset before the plugin even gets to run, thus no JSX expressions means no logs.

On another note, I think I found a bug in your babel plugin implementation:

cool, thanks for the heads up. iirc, I only manually tested it on a file that had one expression in it and there aren't any tests on the plugin yet because I still haven't released the 0.3.x line while I try to discover more processing issues.

[hugmanrique]

I finally found out the cause were the backward slashes (\) and mdx seems to handle the braces fine (sorry for the false statement above, this was already fixed in #145). I've created a short rehype plugin that fixes the issue by double escaping this character:

const visit = require('unist-util-visit');
const remarkMath = require('remark-math');
const katex = require('rehype-katex');

module.exports = require('@zeit/next-mdx')({
  options: {
    mdPlugins: [remarkMath],
    hastPlugins: [
      katex,
      () => tree =>
        visit(tree, 'element', node => {
          if (node.tagName !== 'annotation') {
            return;
          }

          if (!node.properties || node.properties.encoding !== 'application/x-tex') {
            return;
          }

          const { value } = node.children[0];

          // Double escape backward slashes
          node.children[0].value = value.replace(/\\/g, '\\\\');
        })
    ]
  },
  pageExtensions: ['js', 'mdx']
})

The escapes get removed by KaTeX when rendering, but it leaves the original TeX parsed string in an <annotation encoding="application/x-tex"> element.

[ChristopherBiscardi]

The escapes get removed by KaTeX when rendering

I believe this is true for any backslashed content in an MDX file (especially ones that are escape-code related cause trouble). It's also the way I got #366 (comment) working. Wonder if something like this rehype plugin should be included by default?

[johno]

This should now be fixed in 0.20.2.