Markdown versions 3.4.0 and 3.4.1 break documentation build

[kiersten-stokes]

Describe the issue
Markdown released a new version that caused our doc builds to fail both locally and in the CI with:

Extension error (sphinx_markdown_tables):
Handler <function process_tables at 0x7fdd2d294790> for event 'source-read' threw an exception (exception: __init__() missing 1 required positional argument: 'config')

See here for more: https://github.com/elyra-ai/elyra/runs/7362961491?check_suite_focus=true.

Note that Markdown==3.4.1 does not solve this issue. #2836 caps Markdown to v<3.4, but this is a temporary solution.

One potential solution is to remove the doc dependency on sphinx-markdown-tables. When testing locally with sphinx-markdown-tables uninstalled, tables appear to still be rendered correctly for some whereas others see something like the below. Further investigation needs to be done to determine the reason for this and whether this could provide an eventual solution.

[kevin-bates]

@kiersten-stokes - yes, I don't see tables rendered for the trace logging or the GitHub labels (in the previous section) - at least those are the two tables I use as a litmus test.

However, if I replace recommonmark with myst_parser, I get properly rendered tables.

Note that switching to the myst-parser introduces several dozen warnings, the most common of which are:

• WARNING: Non-consecutive header level increase; H3 to H5 [myst.header]
• WARNING: Document headings start at H2, not H1 [myst.header]
• WARNING: 'myst' reference target not found: command-line-interface.html#working-with-pipelines

I believe the header-based warnings can be disabled, although one could argue they should be fixed.

The reference warning is because myst expects the reference to be relative to the source (md) document, not the target (html) document - which also makes IDEs happier (PyCharm will underline the html file stating it is not present) and is generally easier and more natural to deal with.

I think this might be a good time to switch to the myst-parser as it seems more modern and flexible than recommonmark.

[kevin-bates]

Much to my surprise, version 0.0.16 of sphinx-markdown-tables was released yesterday and works with Markdown >= 3.4.1. PR forthcoming to adjust min versions. We can talk about moving to myst-parser another time.