Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for files outside Sphinx project (README.md in parent dir) #7000

Closed
Tracked by #29
NumesSanguis opened this issue Jan 8, 2020 · 19 comments
Closed
Tracked by #29
Labels
type:enhancement enhance or introduce a new feature
Milestone

Comments

@NumesSanguis
Copy link

Is your feature request related to a problem? Please describe.
Many Python repositories have a structure similar to:

project/
  docs/
    conf.py
    index.rst
  src/
  README.md

This README is perfect for writing the project's introduction and some minimal usage examples. GitHub also renders these README's on the landing page (not when in the docs folder). This README is often in the format of Markdown, because in multi-contributer projects, this is the most familiar format.

Good documentation includes an introduction, which the README is perfect for, but currently you cannot include this README in the toctree, because it is outside the docs/ folder.

Describe the solution you'd like
One of the following:

  1. Allow toctree parsing of source files outside the Sphinx project (which allows the extension recommonmark to parse the Markdown formatted README.md)
  2. Allow passing on of a .md file to an extension before trying to parse the file as reST with the .. include:: ../README.md statement.

Describe alternatives you've considered

  1. Use .. include:: ../README.md in a readme_link.rst file (also mentioned in issue: ..include:: statement doesn't parse markdown files correctly #2840).
    Problem: Parses the .md file as reST
  2. Add paths to sys.path in conf.py like: sys.path.insert(0, os.path.abspath('..'))
    Problem: Somehow this doesn't work for me
  3. Use a symlink to the README.md
    Problem: Is not cross-platform (does not work on Windows)
  4. Copy the file on make time with a function.
    Problem: Far from ideal, needlessly making files.
  5. Use M2R with .. mdinclude:: ../README.md; Only option working for me.
    Problem: Using extensions = ['m2r', 'recommonmark'] (or reversed list) gives the error: source_suffix '.md' is already registered in the m2r library.

Additional context

@NumesSanguis NumesSanguis added the type:enhancement enhance or introduce a new feature label Jan 8, 2020
@tk0miya
Copy link
Member

tk0miya commented Jan 8, 2020

+1: Reasonable. README is well known and well used file, and it is usually placed at root of the repository. It would be better to provide a way to include it to main documents (usually placed at directory like docs/).

Note: Sphinx expects all source files are placed under source directory. So it is hard to support external source files. So we have to think how to support it. I thought copying it temporarily is good workaround.

@tk0miya tk0miya added this to the 3.0.0 milestone Jan 8, 2020
@NumesSanguis
Copy link
Author

Copying will likely help many simple cases, but what if the README includes a reference to e.g. data/some_logo.png or other image file? All relative paths in the README will break if you only copy the README.

@tk0miya tk0miya modified the milestones: 3.0.0, 3.1.0 Mar 14, 2020
@mgeier
Copy link
Contributor

mgeier commented Mar 17, 2020

@NumesSanguis There is one alternative that you didn't mention above:

  1. Rename README.md to README.rst, then you can use it in an include directive.

You might see as a disadvantage that you have to use reST syntax in your README.rst, but objectively it's probably not.

@NumesSanguis
Copy link
Author

Thank you for your suggestion @mgeier . That is indeed one way to tackle the issue, but I would argue against that it's objectively not a disadvantage. While obviously I'm familiar with the syntax, that's why I'm using sphinx, it's unfamiliar to many people not / little involved in Python documentation. Markdown is much more common in use, in coding projects of any language. While less advanced, it's faster to write and has a lower barrier of entrance. A quick edit is also done faster.

Therefore, it's quite common that a README.md is already in place before starting with any proper documentation using Sphinx. Having to do 1 adjustment to Sphinx to support Markdown is faster than rewriting the whole README into Sphinx. Markdown is already supported and the only problem here is that it's outside the directory.

So yes, if I absolutely need it, I can rewrite it to .rst, but I think it would help many people if a .md file outside the docs/ folder is supported. Rewriting it to an .rst would put-up an additional barrier for editors unfamiliar with the reST syntax.

ErlendHaa added a commit to ErlendHaa/dlisio that referenced this issue Apr 17, 2020
The release of sphinx 3.0.0 removed some functions that's been
deprecated since v1.4. This made the sphinx-extension 'm2r' break. We
use m2r to include .md files in the sphinx-docs. Although a PR that
seemingly solves the problem is proposed [1], the sole developer of m2r
has not been active in a while. It's unknown when and if this fix will
be merge and even if, having a dependency on a unmaintained project like
m2r makes me feel a bit uneasy.

The ideal action would be to use 'recommonmark' for our .md files.
However, sphinx and recommonmark does not play nice when the .md files
is out of directory, i.e. outside python/docs [2].

That leaves us with two options, either pinning the sphinx version <
3.0.0 and wait for better days, with regards better sphinx/recommonmarks
support, or port the CHANGELOG to rst. As it's unknown whether [2] will
actually see the day of light or not, porting seems most reasonable at
this point in time.

To mention it, there is a bias against using .rst for CHANGELOGs and
READMEs in the community, with the main points being; 'people are
generally more familiar. md' and '.md is easier and faster to write'.
But with a already well-defined structure to our CHANGELOG, the overhead
is minimal.

[1] miyakogi/m2r#55
[2] sphinx-doc/sphinx#7000
ErlendHaa added a commit to ErlendHaa/dlisio that referenced this issue Apr 17, 2020
The release of sphinx 3.0.0 removed some functions that's been
deprecated since v1.4. This made the sphinx-extension 'm2r' break. We
use m2r to include .md files in the sphinx-docs. Although a PR that
seemingly solves the problem is proposed [1], the sole developer of m2r
has not been active in a while. It's unknown when and if this fix will
be merge and even if, having a dependency on a unmaintained project like
m2r makes me feel a bit uneasy.

The ideal action would be to use 'recommonmark' for our .md files.
However, sphinx and recommonmark does not play nice when the .md files
is out of directory, i.e. outside python/docs [2].

That leaves us with two options, either pinning the sphinx version <
3.0.0 and wait for better days, with regards better sphinx/recommonmarks
support, or port the CHANGELOG to rst. As it's unknown whether [2] will
actually see the day of light or not, porting seems most reasonable at
this point in time.

To mention it, there is a bias against using .rst for CHANGELOGs and
READMEs in the community, with the main points being that people are
generally more familiar with .md and that .md is easier and faster to
write.  But with a already well-defined structure to our CHANGELOG, the
overhead is minimal.

[1] miyakogi/m2r#55
[2] sphinx-doc/sphinx#7000
ErlendHaa added a commit to ErlendHaa/dlisio that referenced this issue Apr 17, 2020
The release of sphinx 3.0.0 removed some functions that's been
deprecated since v1.4. This made the sphinx-extension 'm2r' break. We
use m2r to include .md files in the sphinx-docs. Although a PR that
seemingly solves the problem is proposed [1], the sole developer of m2r
has not been active in a while. It's unknown when and if this fix will
be merge and even if, having a dependency on a unmaintained project like
m2r makes me feel a bit uneasy.

The ideal action would be to use 'recommonmark' for our .md files.
However, sphinx and recommonmark does not play nice when the .md files
is out of directory, i.e. outside python/docs [2].

That leaves us with two options, either pinning the sphinx version to
<v3.0.0 and wait for better days, with regards to better
sphinx/recommonmarks support, or port the CHANGELOG to rst. As it's
unknown whether [2] will actually see the day of light or not, porting
seems most reasonable at this point in time.

To mention it, there is a bias against using .rst for CHANGELOGs and
READMEs in the community, with the main points being that people are
generally more familiar with .md and that .md is easier and faster to
write.  But with a already well-defined structure to our CHANGELOG, the
overhead is minimal.

[1] miyakogi/m2r#55
[2] sphinx-doc/sphinx#7000
ErlendHaa added a commit to ErlendHaa/dlisio that referenced this issue Apr 20, 2020
The release of sphinx 3.0.0 removed some functions that's been
deprecated since v1.4. This made the sphinx-extension 'm2r' break. We
use m2r to include .md files in the sphinx-docs. Although a PR that
seemingly solves the problem is proposed [1], the sole developer of m2r
has not been active in a while. It's unknown when and if this fix will
be merge and even if, having a dependency on a unmaintained project like
m2r makes me feel a bit uneasy.

The ideal action would be to use 'recommonmark' for our .md files.
However, sphinx and recommonmark does not play nice when the .md files
is out of directory, i.e. outside python/docs [2].

That leaves us with two options, either pinning the sphinx version to
<v3.0.0 and wait for better days, with regards to better
sphinx/recommonmarks support, or port the CHANGELOG to rst. As it's
unknown whether [2] will actually see the day of light or not, porting
seems most reasonable at this point in time.

To mention it, there is a bias against using .rst for CHANGELOGs and
READMEs in the community, with the main points being that people are
generally more familiar with .md and that .md is easier and faster to
write.  But with a already well-defined structure to our CHANGELOG, the
overhead is minimal.

[1] miyakogi/m2r#55
[2] sphinx-doc/sphinx#7000
@DylanMuir
Copy link

+1 Reasonable request. I am also struggling with this, since m2r seems to be no longer maintained.

@d3an
Copy link

d3an commented Aug 20, 2020

+1 Reasonable request. I am also struggling with this, since m2r seems to be no longer maintained.

I was struggling with the same issue and couldn't even install the unmaintained m2r package with pipenv. However, after running a search on PyPI for alternatives, I discovered an alternative fork of m2r, named m2r2.

The m2r2 package appears to be actively maintained:
GitHub: https://github.com/crossnox/m2r2
PyPI: https://pypi.org/project/m2r2/

By updating the following files, I was able to get a perfectly rendered README.md (from the project root) in Sphinx.

docs/source/conf.py:

extensions = [
    ...
    "m2r2",
]

source_suffix = [".rst", ".md"]

docs/source/readme.rst:

.. mdinclude:: ../../README.md

docs/source/index.rst:

.. toctree::
    :maxdepth: 2
    :caption: Table of Contents:

    readme
    ...

project_structure:

project/
  docs/
    build/
    source/
      conf.py
      index.rst
      readme.rst
  src/
  README.md

Closes #7000

@chrisjsewell
Copy link
Member

chrisjsewell commented Aug 21, 2020

Just use myst-parser and theres no need to have any rST 😉

docs/source/readme.md:

```{include} ../../README.md
```

@chrisjsewell
Copy link
Member

Note in https://myst-parser.readthedocs.io/en/latest/using/howto.html#include-a-file-from-outside-the-docs-folder-like-readme-md

```{include} ../../README.md
:relative-images:
```

Will now re-write local images to the correct path

@maxi-marufo
Copy link

+1 Reasonable request. I am also struggling with this, since m2r seems to be no longer maintained.

I was struggling with the same issue and couldn't even install the unmaintained m2r package with pipenv. However, after running a search on PyPI for alternatives, I discovered an alternative fork of m2r, named m2r2.

The m2r2 package appears to be actively maintained:
GitHub: https://github.com/crossnox/m2r2
PyPI: https://pypi.org/project/m2r2/

By updating the following files, I was able to get a perfectly rendered README.md (from the project root) in Sphinx.

docs/source/conf.py:

extensions = [
    ...
    "m2r2",
]

source_suffix = [".rst", ".md"]

docs/source/readme.rst:

.. mdinclude:: ../../README.md

docs/source/index.rst:

.. toctree::
    :maxdepth: 2
    :caption: Table of Contents:

    readme
    ...

project_structure:

project/
  docs/
    build/
    source/
      conf.py
      index.rst
      readme.rst
  src/
  README.md

Closes #7000

I'm getting this error when building a similar project:

File "/home/max/.local/lib/python3.7/site-packages/sphinx/registry.py", line 266, in add_source_parser for filetype in parser.supported: AttributeError: 'str' object has no attribute 'supported'

The command I'm using is:

sphinx-build -b html docs/source/ docs/build

Any idea?

@tk0miya
Copy link
Member

tk0miya commented Nov 14, 2020

@maxi-marufo It was reported as #8395. And it's a bug of m2r2.

@cstenkamp
Copy link

@chrisjsewell but how do you then include this markdown-page in other rst-pages?
I assumed something like

(readme)=

\```{include} ../../README.md
:relative-images:
\```

and then link it in rst like

:ref:`readme`

would work, but it doesn't. And while myST's documentation explains how to include an rst-page inside an md-page (https://myst-parser.readthedocs.io/en/latest/using/howto.html#include-rst-files-into-a-markdown-file), the opposite is now shown.

@vineetsharma883
Copy link

+1 Reasonable request. I am also struggling with this, since m2r seems to be no longer maintained.

I was struggling with the same issue and couldn't even install the unmaintained m2r package with pipenv. However, after running a search on PyPI for alternatives, I discovered an alternative fork of m2r, named m2r2.

The m2r2 package appears to be actively maintained:
GitHub: https://github.com/crossnox/m2r2
PyPI: https://pypi.org/project/m2r2/

By updating the following files, I was able to get a perfectly rendered README.md (from the project root) in Sphinx.

docs/source/conf.py:

extensions = [
    ...
    "m2r2",
]

source_suffix = [".rst", ".md"]

docs/source/readme.rst:

.. mdinclude:: ../../README.md

docs/source/index.rst:

.. toctree::
    :maxdepth: 2
    :caption: Table of Contents:

    readme
    ...

project_structure:

project/
  docs/
    build/
    source/
      conf.py
      index.rst
      readme.rst
  src/
  README.md

Closes #7000

It worked well for text in README but it did not work on images in my case. @d3an do we need to include some other extra syntax for images ?

@epogrebnyak
Copy link

{include} ../../README.md

With this solition I get WARNING: toctree contains reference to document 'readme_stub' that doesn't have a title: no link will be generated eben though ../../README.md has a top-level header. Am I missing something?

@chrisjsewell
Copy link
Member

With this solition I get WARNING: toctree contains reference to document 'readme_stub' that doesn't have a title: no link will be generated eben though ../../README.md has a top-level header. Am I missing something?

readme_stub need to have its own title:

:title: a title

```{include} ../../README.md
```

@astrojuanlu
Copy link
Contributor

Just came here to say that this solution works like a charm 👍🏽

astrojuanlu added a commit to astrojuanlu/python-idealista that referenced this issue Aug 23, 2021
@cstenkamp
Copy link

@chrisjsewell

README
======
'''{include} ../README.md
:relative-images:
'''

(replace ' by `)
worked for me!

@chrisjsewell
Copy link
Member

p.s. with the latest myst-parser, and docutils>=0.17, you can now include Markdown within RST files, with the parser option:

.. include:: include.md
   :parser: myst_parser.sphinx_

(see https://myst-parser.readthedocs.io/en/latest/sphinx/use.html#include-markdown-files-into-an-rst-file)

@pradyunsg
Copy link
Contributor

@tk0miya What is actionable here?

I see that this is marked for Sphinx 5.0.0 -- but it seems like this support already exists. The request for adding Markdown support is somewhat separate, and already made possible via myst_parser.

Both of the directions that users might want to go is possible now, and it's unclear to me what needs to be addressed here for 5.0.0.

@tk0miya
Copy link
Member

tk0miya commented Jan 12, 2022

Good catch. There are no tasks for this issue. Let's close it!

@tk0miya tk0miya closed this as completed Jan 12, 2022
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 12, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
type:enhancement enhance or introduce a new feature
Projects
None yet
Development

No branches or pull requests