Sync with Doxygen (including LaTeX Equations)

[Codym48]

Outstanding project, @mwouts! I can see this being very useful.

For some other large mixed-language (Python and C++) repositories, we're using Doxygen as the main documentation tool. Any suggestions for the best way to use or update jupytext to support this workflow? I have a test project available at CodyM48/aircraft-motion that describes the problem statement in the README.md:

Attempting to find the best combination of two tools:

• Doxygen for overall project documentation
• Jupyter Notebook for quick interactive visualizations

Given an .ipynb file with LaTeX equations, I paired it with .py and .md files using Jupytext. I manually created a Doxygen C-style comment file and a Doxygen Markdown format file that contain the same content but have been updated to correctly render Doxygen .html pages with LaTeX equations.

Would Jupytext allow us to automatically keep either of these* files in sync?

Synced	File	Description	Diff From Jupytext .md
Manual*	coordinate-frames.dox	Doxygen C-style comment file	See Codym48/aircraft-motion@d94e14c3 commit message
Manual*	coordinate-frames.doxygen.md	Doxygen Markdown format	See Codym48/aircraft-motion@52d6fc13 commit message
Jupytext	coordinate-frames.ipynb	Jupyter Notebook for interaction
Jupytext	coordinate-frames.md	Jupytext (GitHub?) Markdown format
Jupytext	coordinate-frames.py	Python script

Example of page generated by coordinate-frames.doxygen.md:

It seems like the path of least resistance is adding a Doxygen-specific markdown format to jupytext that switches the metadata at the top to use <!-- --> instead of --- --- (or just removes the metadata, which is already possible with default_notebook_metadata_filter) and performs the following find-and-replace operations to match Doxygen LaTeX syntax:

Find	Replace
$$ (start)	\f[
$$ (end)	\f]
$	\f$

Am I on the right path? Would this be valuable?

[mwouts]

Hello @Codym48 , thanks for sharing your use case! That's very interesting.

Let me answer a few points and add a few more questions...

that switches the metadata at the top to use <!-- --> instead of --- ---

That's a great idea! And not only for DOxygen - even on GitHub sometimes we don't want to see the file metadata. Maybe we could offer an additional format option like e.g. comment_notebook_metadata to allow this (note that, since the metadata is in YAML format, I'd prefer to keep the --- ---, and enclose it in the HTML comment, rather than to replace with the HTML comment). And meanwhile, yes, using notebook_metadata_filter="-all" is probably what I would suggest to use!

performs the following find-and-replace operations

What is your expectation on round trips? I mean, if the Doxygen form is found in a notebook, say $\f[$, is it OK to replace it with $$ after a round trip (I mean, we don't go further than applying simple replacements)? Maybe again an additional format option would be the way to go, as you probably want to apply these transformation not only for the Markdown format, but maybe also for the code formats? What would be a good name for the option? Something like doxygen_latex_markers?

And the last question is mine... I see that Doxygen let you use long comments in the code itself, so you'll probably want to the markdown cell of notebooks represented as scripts to have a Doxygen-compatible representation. Do you think that commenting out Markdown cells with ## in Python, and /// in C++, would be the way to do it? Or maybe you don't intend to represent notebooks as scripts for your project?

[Codym48]

that switches the metadata at the top to use <!-- --> instead of --- ---

I'd prefer to keep the --- ---, and enclose it in the HTML comment, rather than to replace with the HTML comment

This seems reasonable to me. Note that there needs to be a blank whitespace line between <!-- and --- and again between --- and --> for Doxygen to parse and render this correctly. Without those whitespace lines, this happens:

For my own information, what's the value of leaving in and commenting out metadata rather than just removing it with notebook_metadata_filter="-all"? Does the answer change if only a subset of the paired files is in version control? (Do we need to store the metadata in .doxygen.md if it's the only one in version control so jupytext knows how to create the .ipynb file from a clean clone?)

performs the following find-and-replace operations

What is your expectation on round trips?

I just confirmed that Jupyter notebooks don't parse the \f$, \f[, and \f] commands, so I can't think of a time when I'd use them within a Notebook:

So I'd support always converting them from those strings in .doxygen.md files to $ and $$ in .ipynb files, even though that means this conversion is not guaranteed to be 1-to-1-to-1 on a round trip. If someone does use the \f[ character sequence in a .ipynb file, it will be left alone on conversion to .doxygen.md and converted to $$ next time the .ipynb file is updated to match the .doxygen.md file.

you'll probably want to the markdown cell of notebooks represented as scripts to have a Doxygen-compatible representation

I like where you're going! I haven't fully thought through the comments-in-Jupyter-notebooks-as-Doxygen-compatible-source-code-comments-in-.py-scripts use case, because it's not one we're using right now. For context, we have source code comments in C++ and Python files, and then we have some additional wrapper or frontpage documentation material captured in .dox and .md files. These files are purely Doxygen documentation and aren't required to compile and run the other source. It's these Doxygen documentation files (.md and .dox) that we'd like to augment with Jupyter notebook capability to make the documentation more interactive and visual, rather than the .py scripts themselves. But let me do some more prototyping of Doxygen-compatible-.py files over in CodyM48/aircraft-motion and I'll get back to you with my findings.

[mwouts]

Great!

So in the short term, we will have two additional format options:

• comment_notebook_metadata
• use_doxygen_latex_markers (or whatever name you prefer, maybe a name focused on doxygen would be better if later on we add more doxygen specificities).

For user friendlyness we could have use_doxygen_latex_markers implying comment_notebook_metadata=True by default.

For my own information, what's the value of leaving in and commenting out metadata rather than just removing it with notebook_metadata_filter="-all"?

As you found out, the metadata that is filtered out is indeed preserved in the .ipynb file. But if you don't distribute the .ipynb file, you may still want to preserve some metadata (by default: the kernel in use in the notebook, and the jupytext format information).

[Codym48]

Ok, I think I actually got a Doxygen-compatible-.py file working using either the grouping into global modules or the member group approach. Basically, any markdown cell would become Doxygen comments for a group that applies to all members (variables, classes, functions, etc.) in the code cells beneath it. I pushed my commits that demonstrate these approaches to Codym48/aircraft-motion. For detailed find-and-replace commands, see:

Approach	Find and Replace Details
global modules	Codym48/aircraft-motion@2e55cf0d
member group	Codym48/aircraft-motion@952432e1

Note that these approaches have some caveats

• assumes the documentation for a specific code cell is placed in a markdown cell above it, not below

• assumes that the documentation in a markdown cell applies to all code cells below it until the next markdown cell

• global modules approach: places all groups at the global level in a Modules page, rather than nesting them under the file or namespace in which they're defined
  

• member group approach: places documentation alongside members in alphabetical order, so no way to view documentation in top-to-bottom order
  

In trying to find the "best" way to do this, I ended up discovering and then deciding not to leverage pandoc and Doxygen python filters like doxypypy and docstrings and many other tools... ultimately, these are two ways to do it with just Doxygen and jupytext, but no guarantee that either is the "best" way.

I think the two additional format options that you list above would be super useful, regardless of if we decide to implement the Doxygen-compatible-.py files.

So: how can I help contribute?

[mwouts]

So: how can I help contribute?

This is very kind of you! Well, I am taking a few days off after the last release, but, sure, I am happy to discuss how we could be working on that...

• for the HTML comment on the metadata header, I'd first code a new test inspired by e.g.

  jupytext/tests/test_header.py

  Lines 126 to 150 in b7cb607

  	def test_multiline_metadata(
  	no_jupytext_version_number,
  	notebook=new_notebook(
  	metadata={
  	"multiline": """A multiline string
  	with a blank line""",
  	"jupytext": {"notebook_metadata_filter": "all"},
  	}
  	),
  	markdown="""---
  	jupyter:
  	jupytext:
  	notebook_metadata_filter: all
  	multiline: 'A multiline string
  	with a blank line'
  	---
  	""",
  	):
  	actual = jupytext.writes(notebook, ".md")
  	compare(actual, markdown)
  	nb2 = jupytext.reads(markdown, ".md")
  	compare(nb2, notebook)

  with markdown changed to

markdown="""<!--

---
jupyter:
  jupytext:
    comment_notebook_metadata: True
---

-->
"""

and then implement the required changes in header.py, and also add the comment_notebook_metadata option in config.py, and to formats._BINARY_FORMAT_OPTIONS. But maybe that is a bit difficult, so if you prefer you can leave this for when I'm back to the project in a few weeks.

• for the latex conversion, we will have to determine where the conversion and its opposite should occur, maybe here (or maybe only in the Markdown specific class?) and here. Also, I am wondering if instead of hard coding the latex to doxygen replacement in jupytext, maybe we could encode the search/replace pattern and pass it as a jupytext option... maybe that would be useful for a few other usecases? What do you think?

[mwouts]

This will come with release 1.6.0.

My expectation is that if you create a jupytext.toml file with this content:

hide_notebook_metadata = true
doxygen_equation_markers = true

then you will get both the hidden metadata header, and the doxygen equation markers.

Cf. this test:

jupytext/tests/test_doxygen.py

Lines 46 to 72 in a13df0d

	def test_doxygen_equation_markers(tmpdir):
	cfg_file = tmpdir.join("jupytext.toml")
	nb_file = tmpdir.join("notebook.ipynb")
	md_file = tmpdir.join("notebook.md")
	cfg_file.write("hide_notebook_metadata = true\ndoxygen_equation_markers = true")
	nb = jupytext.reads(SAMPLE_MARKDOWN, "md")
	del nb.metadata["jupytext"]
	nbformat.write(nb, str(nb_file))
	jupytext_cli([str(nb_file), "--to", "md"])
	md = md_file.read()
	# Doxygen equation markers
	assert "$$" not in md
	assert "\\f[" in md
	assert "\\f]" in md
	assert "\\f$" in md
	assert "\\f$" in md
	# Metadata hidden
	assert md.startswith("<!--\n\n---\n")
	jupytext_cli([str(md_file), "--to", "notebook"])
	nb2 = nbformat.read(str(nb_file), as_version=4)
	compare_notebooks(nb2, nb)

Would you like to give a try to the development version and confirm whether it works as expected?

pip install jupytext==1.6.0rc0