Markdown refactoring?

[reillysiemens]

Hey, @Keats. I've been using Gutenberg for a couple of months to build my personal site and I'm really impressed with the architecture. While working on my site @RadicalZephyr and I encountered an issue. It seems that when arguments are present in a shortcode with a body the paragraph immediately following is not correctly emitted. Unfortunately we have not yet been able to produce a minimal reproduction yet.

This led us to dive into the Markdown parsing code. We found the code pretty challenging to understand and haven't been able to identify what the issue is. One thing we did notice is that it seems like the series of event handlers implicitly specify a state machine. We think it'd be easier to find the bug if we encode the behavior of the numerous mutable state variables as an explicit state machine (possibly multiple state machines). Would you be open to a pull request that restructures the markdown_to_html function to be less monolithic and hopefully easier to reason about?

[Keats]

The markdown rendering is a pile of hacks, I'd be happy if you can make it more manageable!

[Keats]

I released 0.2.2 so you can get started if you want

[Keats]

It would be great to add testcases based on #165 when you get to it

[reillysiemens]

@Keats: Sure thing! I'm planning to spend some time on this this afternoon. 😁

[Keats]

It might be worth looking at https://github.com/kivikakk/comrak as well since pulldown-cmark doesn't really seem maintained. It's about 1.5x-2x slower but it might be ok.

See https://github.com/Keats/markdown-benches-rs for benches

[reillysiemens]

@RadicalZephyr and I have noticed that it's hard to get responses from the pulldown-cmark maintainers, but it would be really worrisome if it was truly unmaintained as rustdoc just removed hoedown in favor of pulldown recently: rust-lang/rust#48274

FWIW, we're continuing to chip away at this every other weekend or so, but it's been slow going because of just how difficult pulldown-cmark's internals are to deal with. The dream of extensible Markdown rendering in Rust hasn't been realized yet... 😅

[Keats]

@RadicalZephyr and I have noticed that it's hard to get responses from the pulldown-cmark maintainers, but it would be really worrisome if it was truly unmaintained as rustdoc just removed hoedown in favor of pulldown recently: rust-lang/rust#48274

I noticed that as well, it got a new release for the Rust project but it doesn't look like other issues are addressed.

FWIW, we're continuing to chip away at this every other weekend or so, but it's been slow going because of just how difficult pulldown-cmark's internals are to deal with. The dream of extensible Markdown rendering in Rust hasn't been realized yet... sweat_smile

Cool, I was actually going to start on it soon if I didn't hear from it soon. I thought with the preprocessing of shortcodes there wasn't too much need for extending pulldown-cmark though, did I miss something? Comrak gives an AST and is actually respecting all aspects of the spec so manipulating the ast should be fairly easier than the events of pulldown-cmark. Maybe add a bench somewhere to see the actual impact on a full article rendering?

[reillysiemens]

With shortcodes being preprocessed that won't be helped by extensible rendering, but I think it's still necessary for anchor tags.

If Comrak gives an AST that sounds preferable. I'll have to check it out! 😁 @RadicalZephyr and I ended up needing to wrap the pulldown-cmark parser in our own lazy AST anyway...