👌 Improve footnote def/ref warnings and translations #931

chrisjsewell · 2024-05-13T01:02:11Z

Footnotes are now parsed similar to the corresponding restructuredtext, in that resolution (between definitions and references) and ordering is now deferred to transforms on the doctree.

This allows for the proper interaction with other docutils/sphinx transforms, including those that perform translations.

In addition, an upstream improvement to unreferenced footnote definitions is also added here: sphinx-doc/sphinx#12730, so that unreferenced and duplicate definitions are correctly warned about, e.g.:

source/index.md:1: WARNING: Footnote [1] is not referenced. [ref.footnote]
source/index.md:4: WARNING: Duplicate footnote definition found for label: 'a' [ref.footnote]

It is of note that warnings for references with no corresponding definitions are deferred to docutils to handle, e.g. for [^a] with no definition:

source/index.md:1: ERROR: Too many autonumbered footnote references: only 0 corresponding footnotes available. [docutils]
source/index.md:1: ERROR: Unknown target name: "a". [docutils]

These warning messages are a little obscure, and it would be ideal that one clear warning was emitted for the issue.
However, it is non-trivial in this extension; to both suppress the existing warnings, and then replace them with a better one,
so for now we don't do it here, and ideally this would be improved upstream in docutils.

closes #930
closes #884
closes #801
closes #445

Add warnings (with source mapping) for unused footnote definitions and footnote references that have no definition.

chrisjsewell · 2024-05-13T01:02:56Z

All good @asmeurer 😄 ?

codecov-commenter · 2024-05-13T01:03:31Z

Codecov Report

Attention: Patch coverage is 92.40506% with 6 lines in your changes missing coverage. Please review.

Project coverage is 90.30%. Comparing base (401e08c) to head (9041bbd).

Files	Patch %	Lines
myst_parser/mdit_to_docutils/transforms.py	86.95%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #931      +/-   ##
==========================================
- Coverage   90.34%   90.30%   -0.04%     
==========================================
  Files          24       24              
  Lines        3469     3507      +38     
==========================================
+ Hits         3134     3167      +33     
- Misses        335      340       +5

Flag	Coverage Δ
pytests	`90.30% <92.40%> (-0.04%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

asmeurer · 2024-05-13T02:31:22Z

I tried this but I got an exception:

Traceback (most recent call last):
  File "/Users/aaronmeurer/miniconda3/envs/ndindex-matplotlib-dev/lib/python3.12/site-packages/sphinx/cmd/build.py", line 298, in build_main
    app.build(args.force_all, args.filenames)
  File "/Users/aaronmeurer/miniconda3/envs/ndindex-matplotlib-dev/lib/python3.12/site-packages/sphinx/application.py", line 355, in build
    self.builder.build_update()
  File "/Users/aaronmeurer/miniconda3/envs/ndindex-matplotlib-dev/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 293, in build_update
    self.build(to_build,
  File "/Users/aaronmeurer/miniconda3/envs/ndindex-matplotlib-dev/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 313, in build
    updated_docnames = set(self.read())
                           ^^^^^^^^^^^
  File "/Users/aaronmeurer/miniconda3/envs/ndindex-matplotlib-dev/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 420, in read
    self._read_serial(docnames)
  File "/Users/aaronmeurer/miniconda3/envs/ndindex-matplotlib-dev/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 441, in _read_serial
    self.read_doc(docname)
  File "/Users/aaronmeurer/miniconda3/envs/ndindex-matplotlib-dev/lib/python3.12/site-packages/sphinx/builders/__init__.py", line 498, in read_doc
    publisher.publish()
  File "/Users/aaronmeurer/miniconda3/envs/ndindex-matplotlib-dev/lib/python3.12/site-packages/docutils/core.py", line 234, in publish
    self.document = self.reader.read(self.source, self.parser,
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aaronmeurer/miniconda3/envs/ndindex-matplotlib-dev/lib/python3.12/site-packages/sphinx/io.py", line 105, in read
    self.parse()
  File "/Users/aaronmeurer/miniconda3/envs/ndindex-matplotlib-dev/lib/python3.12/site-packages/docutils/readers/__init__.py", line 76, in parse
    self.parser.parse(self.input, document)
  File "/Users/aaronmeurer/Documents/MyST-Parser/myst_parser/parsers/sphinx_.py", line 73, in parse
    parser = create_md_parser(config, SphinxRenderer)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aaronmeurer/Documents/MyST-Parser/myst_parser/parsers/mdit.py", line 64, in create_md_parser
    .use(footnote_plugin, inline=False, move_to_end=False, always_match_refs=True)
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aaronmeurer/miniconda3/envs/ndindex-matplotlib-dev/lib/python3.12/site-packages/markdown_it/main.py", line 253, in use
    plugin(self, *params, **options)
TypeError: footnote_plugin() got an unexpected keyword argument 'inline'

chrisjsewell · 2024-05-13T02:32:36Z

I tried this but I got an exception:

you haven't updated your mdit-py-plugins version

asmeurer · 2024-05-13T03:12:30Z

I have the latest version from conda. Is there a dev version I need to use?

chrisjsewell · 2024-05-13T03:14:29Z

I have the latest version from conda. Is there a dev version I need to use?

yeh it was only released on pypi today, so this has only just appeared conda-forge/mdit-py-plugins-feedstock#21

asmeurer · 2024-05-13T06:27:33Z

OK, I was able to get it working. Two things I noticed:

The line number for WARNING: No footnote definition found for label: is wrong. It is giving the line for the beginning of the paragraph instead of the line with the label.
The error message ERROR: Too many autonumbered footnote references: only 0 corresponding footnotes available. is confusing. I'm not sure I understand what it's trying to say. And it seems to be a duplicate error for the same line that gives ERROR: Unknown target name:.

chrisjsewell · 2024-05-13T08:48:35Z

myst_parser/mdit_to_docutils/base.py

+                    # lets remove the footnote references, so that docutils does not produce any more warnings
+                    # we need to replace them with an element though, so that ids can be moved over (otherwise docutils excepts)
+                    if node.get("auto"):
+                        self.document.autofootnote_refs.remove(node)
+                    node.replace_self(nodes.inline(text=f"[^{node['refname']}]"))


code to suppress docutils warnings

chrisjsewell · 2024-05-13T08:52:06Z

tests/test_renderers/fixtures/reporter_warnings.md

+footnote reference with no definition
+.
+[^a]
+.
+<string>:1: (WARNING/2) No footnote definition found for label: 'a' [myst.footnote]
+.


this produced additional docutils warnings, before the fix (https://github.com/executablebooks/MyST-Parser/pull/931/files#r1598103576)

chrisjsewell · 2024-05-13T08:54:53Z

It is giving the line for the beginning of the paragraph instead of the line with the label.

ah well, if you haven't noticed this the same for every "inline" warning in sphinx/docutils (for both restructuredtext and myst)

The error message ERROR: Too many autonumbered footnote references: only 0 corresponding footnotes available. is confusing, ...

these are all from docutils, not myst
but thats strange, because there is specific code to suppress these (see https://github.com/executablebooks/MyST-Parser/pull/931/files#r1598103576) and they don't show up in the tests anymore (https://github.com/executablebooks/MyST-Parser/pull/931/files#r1598109555)

do you have a minimal working example, that could get the test to produce these errors

chrisjsewell · 2024-05-13T09:21:47Z

I definitely get "good" warnings now, if I add some test examples to the top of docs/index.md

asmeurer · 2024-05-13T17:30:41Z

Sorry, don't have time to minimize it, but it's coming from Quansight-Labs/ndindex#136, commit 09541959137694579060a6c48b3e429fa8e30026

/Users/aaronmeurer/Documents/ndindex/docs/indexing-guide/multidimensional-indices/ellipses.md:101: WARNING: No footnote definition found for label: 'tuple-ellipsis-footnote' [myst.footnote]
/Users/aaronmeurer/Documents/ndindex/docs/indexing-guide/multidimensional-indices/tuples.md:403: ERROR: Too many autonumbered footnote references: only 0 corresponding footnotes available.
/Users/aaronmeurer/Documents/ndindex/docs/indexing-guide/multidimensional-indices/tuples.md:403: ERROR: Unknown target name: "integer-scalar-footnote".

asmeurer · 2024-05-13T17:34:25Z

It's possible it's coming from the other cross-reference in the same paragraph. I had to use a workaround to cross-reference cross-page footnotes. These errors all came from me splitting a document into multiple pages and forgetting to move the footnotes.

👌 Improve footnote warnings

b226b53

Add warnings (with source mapping) for unused footnote definitions and footnote references that have no definition.

test and fix issue with footnote reference translations

37c5261

chrisjsewell changed the title ~~👌 Improve footnote definition/reference warnings~~ 👌 Improve footnote definition/reference warnings and translation May 13, 2024

Update test_sphinx_builds.py

4ef3c13

chrisjsewell changed the title ~~👌 Improve footnote definition/reference warnings and translation~~ 👌 Improve footnote def/ref warnings and translations May 13, 2024

chrisjsewell mentioned this pull request May 13, 2024

Footnotes not rendering as expected in localized version #801

Closed

chrisjsewell commented May 13, 2024

View reviewed changes

chrisjsewell added 3 commits May 25, 2024 17:06

Update tox.ini

68b061b

change warning footnote -> footnote_unused

ad36c3d

small code refactor

0961450

jnywong mentioned this pull request Aug 1, 2024

[EPISODE 4] Footnotes do not render in the Spanish version czi-catalystproject/hub-champion-training#172

Open

Merge branch 'master' into footnotes

f0c2dcd

chrisjsewell mentioned this pull request Aug 4, 2024

[rst] Improve unreferenced footnote warnings sphinx-doc/sphinx#12730

Merged

chrisjsewell added 3 commits August 5, 2024 13:25

move footnote ordering to transforms

7a17af9

Update conf.py

9041bbd

Update typography.md

49f288c

chrisjsewell merged commit 850f7c9 into master Aug 5, 2024
17 checks passed

chrisjsewell deleted the footnotes branch August 5, 2024 13:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

👌 Improve footnote def/ref warnings and translations #931

👌 Improve footnote def/ref warnings and translations #931

chrisjsewell commented May 13, 2024 •

edited

Loading

chrisjsewell commented May 13, 2024

codecov-commenter commented May 13, 2024 •

edited by codecov bot

Loading

asmeurer commented May 13, 2024

chrisjsewell commented May 13, 2024 •

edited

Loading

asmeurer commented May 13, 2024

chrisjsewell commented May 13, 2024

asmeurer commented May 13, 2024

chrisjsewell May 13, 2024

chrisjsewell May 13, 2024

chrisjsewell commented May 13, 2024

chrisjsewell commented May 13, 2024

asmeurer commented May 13, 2024 •

edited

Loading

asmeurer commented May 13, 2024

👌 Improve footnote def/ref warnings and translations #931

👌 Improve footnote def/ref warnings and translations #931

Conversation

chrisjsewell commented May 13, 2024 • edited Loading

chrisjsewell commented May 13, 2024

codecov-commenter commented May 13, 2024 • edited by codecov bot Loading

Codecov Report

asmeurer commented May 13, 2024

chrisjsewell commented May 13, 2024 • edited Loading

asmeurer commented May 13, 2024

chrisjsewell commented May 13, 2024

asmeurer commented May 13, 2024

chrisjsewell May 13, 2024

Choose a reason for hiding this comment

chrisjsewell May 13, 2024

Choose a reason for hiding this comment

chrisjsewell commented May 13, 2024

chrisjsewell commented May 13, 2024

asmeurer commented May 13, 2024 • edited Loading

asmeurer commented May 13, 2024

chrisjsewell commented May 13, 2024 •

edited

Loading

codecov-commenter commented May 13, 2024 •

edited by codecov bot

Loading

chrisjsewell commented May 13, 2024 •

edited

Loading

asmeurer commented May 13, 2024 •

edited

Loading