diff --git a/CHANGELOG.md b/CHANGELOG.md index d4e81ca..099224b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,7 @@ - Updated system tests/CI using `pyscaffoldext-custom-extension` - Simplified templates package thanks to `pyscaffold.templates.get_template` - Removed unnecessary `coding: utf-8` comments +- Use symlinks to load `README.md`, `AUTHORS.md`, `CHANGELOG.md` into Sphinx, issues #6, #7. ## Version 0.3.2 diff --git a/README.md b/README.md index 2905986..48552fc 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,5 @@ [![Build Status](https://api.cirrus-ci.com/github/pyscaffold/pyscaffoldext-markdown.svg?branch=master)](https://cirrus-ci.com/github/pyscaffold/pyscaffoldext-markdown) +[![ReadTheDocs](https://readthedocs.org/projects/pyscaffoldext-markdown/badge/?version=latest)](https://pyscaffoldext-markdown.readthedocs.io/) [![Coveralls](https://img.shields.io/coveralls/github/pyscaffold/pyscaffoldext-markdown/master.svg)](https://coveralls.io/r/pyscaffold/pyscaffoldext-markdown) [![PyPI-Server](https://img.shields.io/pypi/v/pyscaffoldext-markdown.svg)](https://pypi.org/project/pyscaffoldext-markdown) [![Downloads](https://pepy.tech/badge/pyscaffoldext-markdown/month)](https://pepy.tech/project/pyscaffoldext-markdown) @@ -7,7 +8,8 @@ # pyscaffoldext-markdown [PyScaffold] extension which replaces [reStructuredText] formatted files -by [Markdown] format except for Sphinx-related files. +by [Markdown] format except for [Sphinx]-related files. + ## Usage @@ -15,14 +17,56 @@ Just install this package with `pip install pyscaffoldext-markdown` and note that `putup -h` shows a new option `--markdown`. Basically this extension will replace `README.rst` by a proper `README.md` and activate the support of Markdown files in Sphinx. -Due to limitations of the Markdown syntax compared to reStructuredText, -the main documentation files still use reStructuredText by default. -Remember to install [wheel] version 0.31 or higher and use [twine] to upload your -package to [PyPI] instead of `python setup.py release` for this to work, e.g.: + +## Limitations + +Due to limitations of the Markdown syntax compared to [reStructuredText], +it is necessary to use symbolic links (and some reStructuredText files) to +avoid keeping multiple copies of files (such as `CHANGELOG.md`) meant to be +placed at the root of the repository but included in the documentation +generated by [Sphinx]. If you are a Windows user please make sure to configure +your system accordingly. The following references might be helpful: + +*Symbolic links on Windows* + +- [http://github.com/git-for-windows/git/wiki/Symbolic-Links](http://github.com/git-for-windows/git/wiki/Symbolic-Links) +- [https://blogs.windows.com/windowsdeveloper/2016/12/02/symlinks-windows-10/](https://blogs.windows.com/windowsdeveloper/2016/12/02/symlinks-windows-10/) +- [https://docs.microsoft.com/en-us/windows/win32/fileio/creating-symbolic-links](https://docs.microsoft.com/en-us/windows/win32/fileio/creating-symbolic-links) + +*Markdown limitations* + +- [https://github.com/readthedocs/recommonmark/issues/191](https://github.com/readthedocs/recommonmark/issues/191) +- [https://github.com/sphinx-doc/sphinx/issues/701](https://github.com/sphinx-doc/sphinx/issues/701) +- [https://github.com/sphinx-doc/sphinx/pull/7739](https://github.com/sphinx-doc/sphinx/pull/7739) + +Windows users that still face problems after configuring their systems for +symbolic links might want to attempt [WSL] or decide to stick with [reStructuredText] +for (problematic) parts of their documentation files. + + +## Building and Releasing + +By default, the [tox] configuration generated by [PyScaffold] is compatible +with Markdown (as implemented in this extension). This means that (after +installing [tox] with [pip] or [pipx]) you can run: + +```bash +tox -e docs # to build your documentation +tox -e publish # to test your project uploads correctly in test.pypi.org +tox -e publish -- --repository pypi # to release your package to PyPI +tox -av # to list all the tasks available +``` + +Please remember that the command `python setup.py release` is no longer +recommended, so if you don't like [tox], please consider using +[Sphinx] and [twine] directly: + ```bash -python setup.py sdist bdist_wheel -twine upload dist/* +python -m pip install -U pip setuptools wheel sphinx twine +python setup.py bdist_wheel # to build your package distributions +make -C docs html # to build your docs +twine upload dist/* # to release your package to PyPI ``` @@ -49,14 +93,18 @@ Please also check PyScaffold's [contribution guidelines]. ## Note -This project has been set up using PyScaffold 3.2. For details and usage -information on PyScaffold see https://pyscaffold.org/. +This project has been set up using PyScaffold 4.0. For details and usage +information on PyScaffold see [https://pyscaffold.org/](https://pyscaffold.org/). [PyScaffold]: https://pyscaffold.org [reStructuredText]: http://docutils.sourceforge.net/rst.html [Markdown]: https://daringfireball.net/projects/markdown/ +[Sphinx]: http://www.sphinx-doc.org/ +[WSL]: https://docs.microsoft.com/en-us/windows/wsl/ +[tox]: https://tox.readthedocs.org/ +[pip]: https://pip.pypa.io/en/stable/ +[pipx]: https://pipxproject.github.io/pipx/ [twine]: https://twine.readthedocs.io/ [PyPI]: https://pypi.org/ -[wheel]: https://wheel.readthedocs.io/ [pre-commit]: http://pre-commit.com/ [contribution guidelines]: https://pyscaffold.org/en/latest/contributing.html diff --git a/docs/conf.py b/docs/conf.py index 258bcab..a7a6f24 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -90,6 +90,7 @@ def setup(app): from recommonmark.transform import AutoStructify params = { + "enable_auto_toc_tree": True, "auto_toc_tree_section": "Contents", "auto_toc_maxdepth": 2, "enable_eval_rst": True, diff --git a/docs/license.rst b/docs/license.rst index 3989c51..c864e69 100644 --- a/docs/license.rst +++ b/docs/license.rst @@ -1,3 +1,9 @@ +.. + Files added to Sphinx' TOC need to have titles, otherwise are ignored. + Since the license is a .txt file, its title is not parsed/recognised. + Therefore, we need a new file with a title and the ``include`` directive. + The ``include`` directive is only available in .rst files. + .. _license: ======= diff --git a/setup.cfg b/setup.cfg index 021c051..21c303b 100644 --- a/setup.cfg +++ b/setup.cfg @@ -47,13 +47,13 @@ package_dir = =src # Require a min/specific Python version (comma-separated conditions) -# python_requires = >=3.8 +python_requires = >=3.6 # Add here dependencies of your project (line-separated) # TODO: Remove conditional dependencies according to `python_requires` above install_requires = importlib-metadata; python_version<"3.8" - pyscaffold>=4.0b2,<5.0a0 + pyscaffold>=4.0b3,<5.0a0 wheel>=0.31 recommonmark diff --git a/src/pyscaffoldext/markdown/extension.py b/src/pyscaffoldext/markdown/extension.py index 41f931d..f3a8e87 100644 --- a/src/pyscaffoldext/markdown/extension.py +++ b/src/pyscaffoldext/markdown/extension.py @@ -1,13 +1,16 @@ """Extension that replaces reStructuredText by Markdown""" -import re -from functools import partial +import os +from functools import partial, reduce +from pathlib import Path from typing import List from configupdater import ConfigUpdater +from pyscaffold import file_system as fs from pyscaffold.actions import Action, ActionParams, ScaffoldOpts, Structure from pyscaffold.extensions import Extension -from pyscaffold.operations import no_overwrite -from pyscaffold.structure import merge, reify_content, resolve_leaf +from pyscaffold.log import logger +from pyscaffold.operations import FileContents, FileOp, no_overwrite +from pyscaffold.structure import merge, reify_leaf, reject from pyscaffold.templates import get_template from . import templates @@ -17,29 +20,6 @@ __license__ = "MIT" -AUTO_STRUCTIFY_CONF = """ -# To configure AutoStructify -def setup(app): - from recommonmark.transform import AutoStructify - - params = { - "auto_toc_tree_section": "Contents", - "enable_eval_rst": True, - "enable_auto_doc_ref": True, - "enable_math": True, - "enable_inline_math": True, - } - app.add_config_value("recommonmark_config", params, True) - app.add_transform(AutoStructify) -""" - -CONV_FILES = { - "README": "readme", - # Use when docutils issue is fixed, see #1 - # "AUTHORS": "authors", - # "CHANGELOG": "changelog" -} - DOC_REQUIREMENTS = ["recommonmark"] template = partial(get_template, relative_to=templates) @@ -51,7 +31,7 @@ class Markdown(Extension): def activate(self, actions: List[Action]) -> List[Action]: """Activate extension. See :obj:`pyscaffold.extension.Extension.activate`.""" actions = self.register(actions, add_doc_requirements) - return self.register(actions, convert_files, before="verify_project_dir") + return self.register(actions, replace_files, before="verify_project_dir") def add_long_desc(content: str) -> str: @@ -70,74 +50,163 @@ def add_long_desc(content: str) -> str: def add_sphinx_md(original: str) -> str: content = original.splitlines() + auto_structify = template("auto_structify").template # raw string # add AutoStructify configuration j = next(i for i, line in enumerate(content) if line.startswith("source_suffix =")) content[j] = "source_suffix = ['.rst', '.md']" - content.insert(j - 1, AUTO_STRUCTIFY_CONF) + content.insert(j - 1, auto_structify) # add recommonmark extension start = next(i for i, line in enumerate(content) if line.startswith("extensions =")) j = next(i for i, line in enumerate(content[start:]) if line.endswith("']")) - content.insert(start + j + 1, "extensions.append('recommonmark')") + content.insert(start + j + 1, 'extensions.append("recommonmark")') return "\n".join(content) def add_doc_requirements(struct: Structure, opts: ScaffoldOpts) -> ActionParams: """In order to build the docs new requirements are necessary now. - This action will make sure ``tox -e docs`` run without problems. + The default ``tox.ini`` generated by PyScaffold should already include + ``-e {toxinidir}/docs/requirements.txt`` in its dependencies. Therefore, + this action will make sure ``tox -e docs`` run without problems. + + It is important to sort the requirements otherwise pre-commit will raise an error + for a newly generated file and that would correspond to a bad user experience. """ + leaf = struct.get("docs", {}).get("requirements.txt") + original, file_op = reify_leaf(leaf, opts) + contents = original or "" - files: Structure = { - "docs": { - "requirements.txt": ("\n".join(DOC_REQUIREMENTS) + "\n", no_overwrite()) - } - } + missing = [req for req in DOC_REQUIREMENTS if req not in contents] + requirements = [*contents.splitlines(), *missing] - original, file_op = resolve_leaf(struct.get("tox.ini")) - original = reify_content(original, opts) - if original: - content = original.splitlines() - j = next(i for i, line in enumerate(content) if "docs/requirements.txt" in line) - content[j] = " -r docs/requirements.txt" - if content[-1]: - content.append("") # ensure empty line at the end (pre-commit) - files["tox.ini"] = ("\n".join(content), file_op) + # It is not trivial to sort the requirements because they include a comment header + j = (i for (i, line) in enumerate(requirements) if line and not is_commented(line)) + comments_end = next(j, 0) # first element of the iterator is a non commented line + comments = requirements[:comments_end] + sorted_requirements = sorted(requirements[comments_end:]) - return merge(struct, files), opts + new_contents = "\n".join([*comments, *sorted_requirements]) + "\n" + # ^ pre-commit requires a new line at the end of the file + files: Structure = {"docs": {"requirements.txt": (new_contents, file_op)}} -def rst2md(content: str) -> str: - """Convert include file from rst to md + return merge(struct, files), opts - Args: - content: content of rst file - Returns: - content of md file - """ - return re.sub(r"(\.\. include:: \.\..+)\.(rst)", r"\1.md", content) +def replace_files(struct: Structure, opts: ScaffoldOpts) -> ActionParams: + """Replace all rst files to proper md and activate Sphinx md. + See :obj:`pyscaffold.actions.Action` + The approach used by recommonmark's own documentation is to include a symbolic link + file inside the docs directory, instead of trying to do a rst's *include*. -def convert_files(struct: Structure, opts: ScaffoldOpts) -> ActionParams: - """Convert all rst files to proper md and activate Sphinx md. - See :obj:`pyscaffold.actions.Action` + References: + - https://github.com/readthedocs/recommonmark/issues/191 + - https://github.com/sphinx-doc/sphinx/issues/701 + - https://github.com/sphinx-doc/sphinx/pull/7739 """ - struct = struct.copy() - for file, template_name in CONV_FILES.items(): - # remove rst file - _, file_op = resolve_leaf(struct.pop(f"{file}.rst")) - # add md file - struct[f"{file}.md"] = (template(template_name), file_op) + # Remove all unnecessary .rst files from struct + unnecessary = [ + "README.rst", + "AUTHORS.rst", + "CHANGELOG.rst", + "docs/index.rst", + "docs/readme.rst", + "docs/authors.rst", + "docs/changelog.rst", + ] + struct = reduce(reject, unnecessary, struct) + content, file_op = reify_leaf(struct["setup.cfg"], opts) + struct["setup.cfg"] = (add_long_desc(content), file_op) + + docs = struct.pop("docs", {}) # see comments on ``files`` + content, file_op = reify_leaf(docs["conf.py"], opts) + root = Path(opts.get("project_path", ".")) + + # Define replacement files/links + files: Structure = { + "README.md": (template("readme"), no_overwrite()), + "AUTHORS.md": (template("authors"), no_overwrite()), + "CHANGELOG.md": (template("changelog"), no_overwrite()), + "docs": { + **docs, + # by popping the docs and merging them back we guarantee the '*.md' + # files at the root of the repository are processed first, then when it is + # time to process the `docs` folder, they already exist and can be symlinked + "conf.py": (add_sphinx_md(content), file_op), + "index.md": (template("index"), no_overwrite()), + "readme.md": (None, no_overwrite(symlink(root / "README.md"))), + "authors.md": (None, no_overwrite(symlink(root / "AUTHORS.md"))), + "changelog.md": (None, no_overwrite(symlink(root / "CHANGELOG.md"))), + }, + } - content, file_op = resolve_leaf(struct["setup.cfg"]) - struct["setup.cfg"] = (add_long_desc(reify_content(content, opts)), file_op) + return merge(struct, files), opts - # use when docutils issue is fixed, see #1 - # for file in ("authors", "changelog"): - # content, file_op = resolve_leaf(struct["docs"].pop(f"{file}.rst")) - # struct["docs"][f"{file}.md"] = (rst2md(reify_content(content, opts)), file_op) - content, file_op = resolve_leaf(struct["docs"]["conf.py"]) - struct["docs"]["conf.py"] = (add_sphinx_md(reify_content(content, opts)), file_op) +def is_commented(line): + return line.strip().startswith("#") + + +def symlink(original_file: fs.PathLike) -> FileOp: + """Returns a file operation that creates a symlink to ``original_file``.""" + # TODO: Transfer this function to PyScaffold's core (and split it into 2: + # a file_system.symlink and an operations.symlink) + + def _symlink(path: Path, _: FileContents, opts: ScaffoldOpts): + """See ``pyscaffoldext.markdown.extension.symlink``""" + should_pretend = opts.get("pretend") + should_log = opts.get("log", should_pretend) + # ^ When pretending, automatically output logs + # (after all, this is the primary purpose of pretending) + + if should_log: + logger.report("symlink", path, target=original_file) + + if should_pretend: + return path + + # Since errors in Windows can be tricky, let's print meaningful messages + if path.exists(): + if opts.get("force"): + path.unlink() + else: + raise FileExistsError( + "Impossible to create a symbolic link " + f"{{{path} => {original_file}}}.\n{path} already exist.\n" + ) + + if not Path(original_file).exists(): + raise FileNotFoundError( + "Impossible to create a symbolic link " + f"{{{path} => {original_file}}}: {original_file} does not exist" + ) + + try: + # Relative links in Python might be tricky + # the following implementation is very difficult to be replaced by something + # completely pathlib-based, see https://bugs.python.org/issue37019 + os.symlink(os.path.relpath(original_file, path.parent), path) + return path + except OSError as ex: + raise SymlinkError(path, original_file) from ex + + return _symlink + + +class SymlinkError(OSError): + """\ + Impossible to create a symbolic link {{{link_path} => {original_file}}}. + If you are using a non-POSIX operating system, please make sure that your user have + the correct rights and that your system is correctly configured. + + Please check the following references: + http://github.com/git-for-windows/git/wiki/Symbolic-Links + https://blogs.windows.com/windowsdeveloper/2016/12/02/symlinks-windows-10/ + https://docs.microsoft.com/en-us/windows/win32/fileio/creating-symbolic-links + """ - return struct, opts + def __init__(self, link_path, original_file, *args, **kwargs): + docs = self.__class__.__doc__ or "" + msg = docs.format(original_file=original_file, link_path=link_path) + super().__init__(msg, *args, **kwargs) diff --git a/src/pyscaffoldext/markdown/templates/auto_structify.template b/src/pyscaffoldext/markdown/templates/auto_structify.template new file mode 100644 index 0000000..dd5d36f --- /dev/null +++ b/src/pyscaffoldext/markdown/templates/auto_structify.template @@ -0,0 +1,15 @@ +# Configure AutoStructify +# https://recommonmark.readthedocs.io/en/latest/auto_structify.html +def setup(app): + from recommonmark.transform import AutoStructify + + params = { + "enable_auto_toc_tree": True, + "auto_toc_tree_section": "Contents", + "auto_toc_maxdepth": 2, + "enable_eval_rst": True, + "enable_math": True, + "enable_inline_math": True, + } + app.add_config_value("recommonmark_config", params, True) + app.add_transform(AutoStructify) diff --git a/src/pyscaffoldext/markdown/templates/changelog.template b/src/pyscaffoldext/markdown/templates/changelog.template index e215977..205cc5e 100644 --- a/src/pyscaffoldext/markdown/templates/changelog.template +++ b/src/pyscaffoldext/markdown/templates/changelog.template @@ -1,6 +1,6 @@ # Changelog -## Version 0.1 +## Version 0.1 (development) - Feature A added - FIX: nasty bug #1729 fixed diff --git a/src/pyscaffoldext/markdown/templates/index.template b/src/pyscaffoldext/markdown/templates/index.template new file mode 100644 index 0000000..8e73726 --- /dev/null +++ b/src/pyscaffoldext/markdown/templates/index.template @@ -0,0 +1,37 @@ +# ${name} + +${description} + + +## Note + +> This is the main page of your project's [Sphinx] documentation. It is +> formatted in [Markdown]. Add additional pages by creating md-files in +> `docs` or rst-files (formated in [reStructuredText]) and adding links to +> them in the `Contents` section below. +> +> Please check [Sphinx], [recommonmark] and [autostructify] for more information +> about how to document your project and how to configure your preferences. + + +## Contents + +* [Quickstart](readme) +* [License](license) +* [Authors](authors) +* [Changelog](changelog) +* [Module Reference](api/modules) + + +## Indices and tables + +```eval_rst +* :ref:`genindex` +* :ref:`modindex` +* :ref:`search` +``` + +[Sphinx]: http://www.sphinx-doc.org/ +[Markdown]: https://daringfireball.net/projects/markdown/ +[reStructuredText]: http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html +[recommonmark]: https://recommonmark.readthedocs.io/en/latest diff --git a/src/pyscaffoldext/markdown/templates/readme.template b/src/pyscaffoldext/markdown/templates/readme.template index c250a32..3042d09 100644 --- a/src/pyscaffoldext/markdown/templates/readme.template +++ b/src/pyscaffoldext/markdown/templates/readme.template @@ -1,4 +1,4 @@ -# ${project} +# ${name} ${description} diff --git a/tests/test_markdown.py b/tests/test_markdown.py index 21696d2..8c40c55 100644 --- a/tests/test_markdown.py +++ b/tests/test_markdown.py @@ -4,7 +4,17 @@ from pyscaffold import __version__ as pyscaffold_version from pyscaffold import api, cli -from pyscaffoldext.markdown.extension import CONV_FILES, Markdown +from pyscaffoldext.markdown.extension import DOC_REQUIREMENTS, Markdown + +CONV_FILES = [ + "README", + "AUTHORS", + "CHANGELOG", + "docs/index", + "docs/readme", + "docs/authors", + "docs/changelog", +] @pytest.mark.slow @@ -21,18 +31,24 @@ def test_create_project_with_markdown(tmpfolder): # when the project is created, api.create_project(opts) + assert (tmpfolder / "proj/docs").is_dir() - # then markdown files should exist + # then markdown files should exist, for file in CONV_FILES: assert (tmpfolder / f"proj/{file}.md").exists() assert not (tmpfolder / f"proj/{file}.rst").exists() - # and the content-type of README should be changed accordingly - existing_setup = (tmpfolder / "proj" / "setup.cfg").read_text() + # the content-type of README should be changed accordingly, + existing_setup = (tmpfolder / "proj/setup.cfg").read_text() cfg = ConfigParser() cfg.read_string(existing_setup) assert "text/markdown" in str(cfg["metadata"]["long-description-content-type"]) + # and new doc requirements should be added to docs/requirements.txt + requirements_txt = (tmpfolder / "proj/docs/requirements.txt").read_text() + for req in DOC_REQUIREMENTS: + assert req in requirements_txt + @pytest.mark.slow @pytest.mark.system