Javascript implementation of MyST markdown

[choldgraf]

I can't remember if we've created an issue to track this or not, so I figured we should start one here to keep track of relevant information etc.

Background - currently MyST markdown exists as a collection of extensions to a Python markdown parser that we've developed called markdown-it-py. This was ported from the popular and extensible Javascript MD parser called markdown-it.

The goal behind this was to have a Python implementation of markdown-it that was structured very similarly, in the hopes that it would be easy to port syntax extensions back and forth between JS and Python. This might also be a path towards getting MyST markdown support in JupyterLab, if it also adopts markdown-it for its parser.

ToDo: At a minimum, we could:

• Write a set of markdown-it extensions that mimic the extra syntax added by MyST-markdown. I believe those elements are in the myst-blocks and myst-role folders.
• Write the writers that will take the tokens generated from the initial parse and output HTML elements (and whatever else needs to be done to get the "output" side of the markdown-it extensions working).
• Write up some kind of javascripty wrapper that uses markdown-it to parse markdown and outputs rendered tokens. This could be the "MyST Markdown javascript library"
• Think about ways to incorporate that library into other javascript applications, such as:
  • A Jupyter Notebook extension that over-rides the markdown parser with our parser
  • A Jupyter lab extension that over-rides the markdown parser with our parser
  • Similar extensions for other tools (e.g. iooxa?)

Note that some of this has already been done in the MyST markdown vscode extension here: https://github.com/executablebooks/myst-language-support/blob/master/src/md_it_plugin.ts?rgh-link-date=2020-06-24T01%3A07%3A08Z

[chrisjsewell]

Erm you know the vs code extension already does some of this: https://github.com/executablebooks/myst-language-support/blob/master/src/md_it_plugin.ts

[choldgraf]

nope, I didn't - added a note about it to the list above.

[choldgraf]

(and feel free to correct, add, or edit anything that might not be right above...I mostly just wanted to have a place where we can track all of this stuff since it's kind-of split across a few repositories)

[choldgraf]

just a note here - it looks like someone is working on a Jupyter Lab extension that makes Lab use markdown-it, may be useful for this: https://github.com/agoose77/jupyterlab-markup

[rowanc1]

Starting to dig into this a bit more, I think the goal is to have a markdown-it-myst library, similar naming conventions to others (e.g. markdown-it-texmath)?

The library would provide the .ts port of the myst-blocks and myst-role folders.

My understanding in the beginning would be that this focuses mostly on the parsing rather than a "functional" rendering, as the current myst.py is supported by sphinx to do the rendering?

Also looking through the vs-code extension I am not quite sure how mature the plugin and/or how related to just fixing the parsing in vs-code? e.g. line#122 looks like it doesn't capture all of the roles/directives?

I have started some work in this direction over in https://github.com/iooxa/editor and will continue to hack around trying to get some roles/directives parsing based on the python extensions.

[choldgraf]

Yeah I think that we may want support for some roles and directives that are common (e.g., note, warning) as well as the other extended syntax pieces, but we can't support everything that could be in Sphinx. I think this means that we'll need to define a "minimal set" of roles/directives supported by MyST so that people know to expect certain kinds of behavior.

[rowanc1]

I think one of the pieces that I am building over in iooxa is actually an alternative renderer for sphinx like cross-refs, math, TOC, etc. but much less heavy on the python integrations (code documentation). So that could be a good cross over for alternative rendering environment (albeit out of scope completely for this first library).

[choldgraf]

yeah - in a jupyter / vscode / iooxa environment we'd need to have our own markdown-it-myst rendering rules that didn't use Sphinx...+1 on starting out with iooxa and seeing how it looks! We can always create non-iooxa ports if it makes sense to do so

[chrisjsewell]

Yeh cool @rowanc1 keep us posted how it goes 😄 and ideally perhaps there would be some shared npm package between the VS Code extension, iooxa, etc, defining some frontend-agnostic code

[choldgraf]

yeah totally - I feel like we want something that'll output a reliable HTML blob that can be styled in many other ways depending on the interface

[rowanc1]

Ok, took an initial crack at roles. It is in a folder without anything special in it -- I will continue to iterate a bit there and at some point we can spin it out into a package.

Right now, I am tracking {math}`Ax=b` and {abbr}`CSS (Cascading Style Sheets)`, which turns into a span class="math" and title-ed abbr respectively. See the tests. It is a start ... ! 🚀

It would be interesting to get a handle on what are the most sensible things in your mind to have an independent html renderer for (and also how dependent on javascript or not the base render is) ... and what we give up on (e.g. cross-refs?).

Tomorrow I will sink a bit of time into the blocks. Should go a bit faster now that I am more familiar with all the libraries!

[rowanc1]

Wondering if either @chrisjsewell or @choldgraf could give me a pointer on how you deal with block directives on the python side?

Are these deferred to a later renderer? For example:

```{code-block} python
:lineno-start: 10
:emphasize-lines: 1, 3

a = 2
print('my 1st line')
print(f'my {a}nd line')
```

Is this parsed in markdown-it as a fence, with Token.info and Token.content, and then later in the sphinx side cleaned up?

I am looking into this on the javascript side, and thinking it might be nice to know about this earlier on. Any thoughts or pointers are appreciated!

[choldgraf]

Indeed I believe that you are right - if you look at the myst-parser's docutils renderer (which is the one that spits out Sphinx objects), it looks like the directive rendering is triggered from within the code fence rendering if the "language" starts and ends with { }:

https://github.com/executablebooks/MyST-Parser/blob/master/myst_parser/docutils_renderer.py#L341

and director rendering code here:

https://github.com/executablebooks/MyST-Parser/blob/3a3cd434c22f2e9bb1ad57c74cf534c6b8eec627/myst_parser/docutils_renderer.py#L625

[chrisjsewell]

Are these deferred to a later renderer... Is this parsed in markdown-it as a fence?

Yes and yes: https://github.com/executablebooks/MyST-Parser/blob/3a3cd434c22f2e9bb1ad57c74cf534c6b8eec627/myst_parser/docutils_renderer.py#L341-L342

I just didn't see much mileage (at least for myst-parser) in specifically designating the tokens as directives, since all you need to do is loop through the tokens and check which Token.content has braces.

If you are looking into this, you may also wish to note the nest_tokens function which is a markdown-it-py only addition, which I added to make the tokens more compatible with the sphinx rendering process.
(I should probably add this to the port.yaml)

[choldgraf]

wohooo I beat you by seconds 😅

[chrisjsewell]

wohooo I beat you by seconds

slow and steady wins the race 😉 🐢

[rowanc1]

Awesome. Thank you!

In looking through this on my end, the most useful place for me to put things would be in the parse/tokenizer stage, and start to make some of the sphinx extensions explicit. For example, the admonitions (note, warn etc.), and likely some other ones here.

My workflow at least will be to parse with markdownit, and then transfer the tokens to another format, completely skipping the render step, which is where the code for directives lives currently.

Actually, @chrisjsewell is that what this code is doing:
https://github.com/executablebooks/myst-language-support/blob/master/src/md_it_plugin.ts#L122

[choldgraf]

@rowanc1 I agree - I think we could all agree on a minimal set of directives that should exist across different renderer implementations. In the myst-parser we just farm that out to "whatever is in Sphinx" but for a javascript HTML renderer we don't have something like that so we should choose carefully what to start with and document it clearly so that users aren't confused (e.g. "why does {jupyter-execute} work in myst-parser but not markdown-it-myst???). Perhaps we could create an issue explicitly to brainstorm this?

[chrisjsewell]

Actually, @chrisjsewell is that what this code is doing

That is where I currently address recursive parsing of admonition directives (i.e. parsing their contents as markdown). But note, I would like to look into essentially replacing this (for python and JS) with https://github.com/markdown-it/markdown-it-container (see https://github.com/executablebooks/markdown-it-py/issues/23 and executablebooks/MyST-Parser#154)

[rowanc1]

Ok, have added some initial work using the markdown-it-container, it should be setup to strip out the metadata/yaml and add that to the container attrs. Right now it works for admonitions, but should be relatively easy to extend to other directives.

I will continue to clean it up wrap with tests etc. in the next day or so.

[chrisjsewell]

👍

but should be relatively easy to extend to other directives

Although it can only be used for admonition-like directives, i.e. directives where the content it interpreted specifically as markdown

[choldgraf]

I think we might want it for some other authoring-focused directives like figure (but again, we should probably just have a specification where we define this minimal set)

[chrisjsewell]

well yes because, because that fits my description of "the content it interpreted specifically as markdown", which in this case is the caption. Theres not too many other directives I can think of that do though

[chrisjsewell]

Obviously the question arises with these as to how you parse directive options, and that's where a (new) markdown-it-fieldlist extension would probably come in to process similar to https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#field-lists
(my only misgiving of this syntax would be the clash with the Github Flavoured Markdown's emoji syntax)

[rowanc1]

I have put together a repo here:
https://github.com/iooxa/markdown-it-myst

This is starting to get a bit better around the edges, but still needs quite a bit of work. Feel free to look at the code if you are so inclined.

I put together an issue here: jupyter-book/mystmd#1 that shows the parser in action - right now it is pulling from the test cases which you can see in the fixtures folder. That is probably the easiest way to see where I am at. The only undocumented piece is math at the moment.

I have done a first pass at internal references, which is nice and fun to have some capabilities outside of the sphinx ecosystem.

I am probably not going to get much work done on this the rest of the week - aiming to pick it up again around the 29th.

Let me know what you think!

Demo

[choldgraf]

I think this looks awesome! thanks for putting in the work to prototype this out :-)

One question: is your plan to keep this in iooxa or to potentially donate it to executablebooks?

[rowanc1]

Yeah, would like to move it across at some point. Probably when it is a tiny bit more mature, I am pushing straight to master as I flush it out currently etc.

I would also love to be a maintainer on the EB side when it moves, if possible. Looking forward to chatting about how that would work.

[choldgraf]

I would also love to be a maintainer on the EB side when it moves, if possible. Looking forward to chatting about how that would work.

well the reason that I was asking was to see if you'd prefer to just have this live in EBP and get admin rights so you can do as you wish with the repository. This is what we've done for other repos - so long as the README makes it clear what users should expect, I think it's totally fine. I guess the only concern would be if you and @chrisjsewell had major unresolvable differences in opinion for how things should be implemented/designed, but I don't detect any of that right now

That said - I'm happy to go with whatever pattern works best for you!

[rowanc1]

If you are ok with it being incubated over in EBP, then I can clear up the readme, letting folks know it is still in alpha, etc and get it ready to move over. (Probably won't get to it til mid next week)

My biggest worry was that I would have to stop using so many emoji in my commit messages. But that has been firmly put to rest. :)

[chrisjsewell]

My biggest worry was that I would have to stop using so many emoji in my commit messages.

Yep, to become an EBP member, you just have to make signed declaration to adhere to the emoji convention 😆 https://executablebooks.org/en/latest/dev-conventions.html#commit-messages which I see you've already done 👍
(and @choldgraf has already been flouting 👀 😜 jupyter-book/jupyter-book@93bfecf)

On this topic I was actually thinking to add some more guidance for JS/TS best practices at some point:

1. I note that you use yarn as opposed to npm for package management. I've been using npm, and from what I gather there is not too much difference these days (https://afteracademy.com/blog/npm-vs-yarn). Any specific pros for using yarn?
2. Test frameworks in JS/TS are more fractured than the default pytest in Python. You are using jest for testing. Before I have used mocha+chai (e.g. in https://github.com/executablebooks/myst-language-support) and in Add regexp prompt matching (+ style and testing) sphinx-copybutton#82 I took the opportunity to review frameworks (e.g. https://raygun.com/blog/javascript-unit-testing-frameworks/) and found ava to be a nice minimalistic framework.

Neither are deal breakers, but interested to hear your opinion?

[rowanc1]

Hey @chrisjsewell - @choldgraf and I managed to figure out the transfer of the repo, so this is now an official executablebooks project 🎉

I switched over to yarn due to speed and its work, installing flat dependencies (annoying in some of the repos I have dealt with lately if there are multiple versions), as well as local-caching of the repos. I also quite like the publish workflow in yarn better than npm. They are inter-changeable though, so it shouldn't force anything on users.

For test frameworks, I think that I have just inherited jest because of backend server testing that uses supertest / jest. I don't have a whole lot of experience in other frameworks or much to add here! If you have strong opinions I am happy to migrate towards something else.

I have done an initial publish to npm of the repo, and over the next few weeks it would be good to get that under CD, as well as get other people onto being owners of the package on NPM.

Would it be worth transferring this issue over to the markdown-it-myst repo? And/or closing and starting a more focused thread over there?

[choldgraf]

btw - not strictly related to this issue but I think the folks in here may be interested in this "vision for jupyterlab" conversation here: jupyterlab/frontends-team-compass#80

[choldgraf]

Would it be worth transferring this issue over to the markdown-it-myst repo? And/or closing and starting a more focused thread over there?

now that we have markdown-it-myst in the org: https://github.com/executablebooks/markdown-it-myst I'd be +1 on closing this and using issues / roadmaps / etc on that project to guide this thread further

[rowanc1]

Thanks for the link over to the roadmap conversation for JupyterLab. Interesting stuff and will be cool to see where it filters down to!

Feel free to close when you want (I don't have permission on this repo). And we can pick up specific conversations on the other repo!

[choldgraf]

sounds good 👍