Use setuptools_scm to package version number into metadata

[2bndy5]

resolves #714 and closes #746

setuptools_scm repo (for reference)
python docs on importlib.metadata module as that is now the preferred implementation to get version info for a specified (& installed) package.

This replaces the need to store the version number in the package's __init__.py module. Instead the version information can be found within the installed package metadata (as recommended by PEP 440). I've additionally configured the setuptools_scm module to generate a version.py file in the breathe folder for distibution.

This switch also reduces the need to use git describe in the docs' conf.py to get the git tag. Thus, I've simplified that process as well. While I was in the conf.py file, I noticed that the mathjax_path option was outdated (per info in sphinx-docs), so I updated that as well. See the docs' CI artifact to verify math formulas are correctly rendered (as I have already done locally).

[michaeljones]

Looks great. Thank you for taking the time.

I don't have the detailed knowledge of the python ecosystem to fully follow everything. I appreciate the links you provided. It gives me confidence that it is the right thing, even if I don't have the patience to read all the documents :)

Happy to see us using the pyproject.toml more.

[michaeljones]

Why the . out of interest?

[2bndy5]

conf.py now gets the version info from breathe as an installed package. That's what the dot does. This should not affect RTD as the readthedocs.yaml already tells RTD to install breathe as well.

[michaeljones]

Does this change the release process at all? Not that I'm sure what it is.

[2bndy5]

local testing revealed that the package metadata (used from this PR's changes) did not output a v prefix.

[vermeeren]

Presumably get_version already strips that internally? Breathe has used the v prefix for git tags for a long, long time now.

[2bndy5]

Does this change the release process at all? Not that I'm sure what it is.

pinging @vermeeren because git blame says the mkrelease makefile originated from them. Looking at that file briefly, I don't think the release process should be affected at all by this PR. My only concern would be the prefixed "v" in the tag pushed (see this review comment).

IMHO, the entire release process could be done via Github CI workflow, but that would be a separate issue as it would require @michaeljones (or whoever has access to the pypi account) to create a secret token on this repo's settings that can be used instead of Melvin's PGP token...

local testing

(venv) PS C:\Users\ytreh\Documents\GitHub\breathe> pip install .
Successfully installed breathe-4.31.1.dev41+g715746f
(venv) PS C:\Users\ytreh\Documents\GitHub\breathe> python -c "from importlib.metadata import version; print(version('breathe'))"
4.31.1.dev41+g715746f

Contents of breathe/version.py (generated from setuptools_scm module)

# coding: utf-8
# file generated by setuptools_scm
# don't change, don't track in version control
version = '4.31.1.dev41+g715746f'
version_tuple = (4, 31, 1, 'dev41', 'g715746f')

[vermeeren]

Presumably get_version already strips that internally? Breathe has used the v prefix for git tags for a long, long time now.

[vermeeren]

@2bndy5 As far as I know this determines the version seen on PyPI with how the packaging scripts runs, but I am no way an expert on this. Are you sure removing this line won't cause issues?

I think you can verify by tagging a fake version locally (a to-be 4.32.0) and then running ./mkrelease pack from repo root. That'll dump what is normally uploaded in mkrelease_tmp/dist/*.wlh. Inside that there should be some metadata file iirc.

[vermeeren]

I just read #714 (comment), so presumably this does work rightly with current version.

After this review round I think it's time to just test this and if it works it works.

[2bndy5]

EDIT: This line is replaced by the usage of the setuptools_scm pkg at build time. Sorry I didn't realize which file this comment corresponded to.

This line has nothing to do with pypi. It was only ever a convenience for users to programmatically get the extension's version tag. According to PEP440 (link in PR description), users are encouraged to use

>>> from importlib.metadata import version  
>>> version('wheel')  
'0.32.3'

(snippet from pydocs - link in PR description)

---

I was wondering how to use the mkrelease script. I'll try this out, thanks

[vermeeren]

This is also a reply to #766 (comment) below.

@2bndy5 Ah yeah, the mkrelease utility does rely on a POSIX-compliant environment, so in practice will need some compatibility tooling on Windows.

The way you tagged it does actually seem to be correct and the script also seems to execute correctly producing exactly the expected output. So in that case the version number isn't being detected properly by the python tooling with the new method, which is probably related to its absolute removal I mentioned here.

Effectively mkrelease is fairly simple and was added to automate some stuff I used to do manually every single time, the pack function is actually very simple and can be fully found here https://github.com/michaeljones/breathe/blob/master/mkrelease#L29.

I think the problem is that a "git archive" tarball, either via CLI or download from a GitHub tag, does not contain actual git information (i.e. .git directory) So when the python3 setup.py sdist bdist_wheel is called it's called from a git-less, clean unpacked Breathe archive as-if downloaded from GitHub from the tag. git tag does then not work from there.

With this realisation I'm not even sure if you can truly have fully automated versioning as it has to work without the git history, meaning the current version has be stored outside of git at at least one location. Linux/BSD distributions usually package based upon the source tarball + PGP signature, not the full git source (to avoid big size and for archiving), so I don't see an easy way out of this.

Could you share thoughts about this? I'm not sure how to cleanly solve this at this point.

[2bndy5]

@vermeeren I think setup.py has been stripping the "v" prefix automatically for some time. You should see a warning about this when running python3 setup.py <build/install> (warning may not appear when using pip install . at repo root).

[2bndy5]

Not sure if I did this right:

git tag v4.32.0
./mkrelease pack

which resulted in folder structure like so:

It seems the tag is properly used in the pkg built but not in the bash script's prep. I was able to verify the proper tag in the pkg's metadata file ✔️ . It may be worth noting that I typically don't use git bash to push tags.

---

IMO, this entire process should be done automatically via a github CD workflow on a "published realease" event. Using a Linux-only mechanism tends to limit dev's options for cross-platform software (ie the repo's root Makefile would be better served as a python script - maybe even using tox pkg).

[vermeeren]

@2bndy5 As for automation, I kind of agree but it would mean you cannot securely handle the PGP signatures. With the current setup we have a two-factor security setup for distributions packaging Breathe. So if somehow someone malicious took over the Breathe repository (or if GitHub itself got hacked), such a release could not be packaged as it is not signed by my PGP key, which I only have locally on my own devices.

It is definitely an inconvenience though but here I feel like it's a trade-off worth having in this case.

[2bndy5]

Honestly, we've hit a separate issue worthy of discussion (perhaps in a new thread titled auto-publish via github CD).

I'm not even sure if you can truly have fully automated versioning as it has to work without the git history

I didn't think building the extension from a .git-less source is a huge priority. I do think the docs' quickstart should be updated to demonstrate using the extension from a pip install as that is the norm/standard.

I think the problem is that a "git archive" tarball, either via CLI or download from a GitHub tag, does not contain actual git information (i.e. .git directory) So when the python3 setup.py sdist bdist_wheel is called it's called from a git-less, clean unpacked Breathe archive as-if downloaded from GitHub from the tag

If someone is building from source, then it would be best to do so from a fork/clone (not from release assets). I've never heard of this approach (using github release assets) being preferred by any CPython pkg; it feels like you're trying to ignore conveniences (like pip) provided by python itself.

• If one needs to use non-traditional install locations, then that's what the venv module (std lib) is for.

Linux/BSD distributions usually package based upon the source tarball + PGP signature, not the full git source

This distributed software isn't linux-only, but now I understand where you're coming from using the PGP signature. TBH, I don't know much about signing/verifying PGP keys/signatures (or anything of that nature).

• Is the PGP signature actually useful to either pypi or end-users? It sounds like one would have to go out of their way to make use of this PGP signature (let alone identify if a pkg is PGP signed).

python setup.py now invokes pip, which is the real problem behind #714 due to enacting PEP517). Remember, the distributed package is automatically slimmed down to only what is needed to run (as specified in setup.py), so there's no concerns about the uploading git-related source material to pypi.

With the current setup we have a two-factor security setup for distributions packaging Breathe

I also have 2FA enabled for my pypi account. When I use a github CD workflow to auto-publish to pypi (on a "created release" event), I have a pkg-specific secret token in the repo's settings. This can be setup using the pypi account (probably done by @michaeljones ), and it is the preferred method of authenticating uploaded distributions. I can give more hints (possibly another PR or augment this one), but you (or another admin) need to know about creating github secret tokens and using them within github action workflows.

or if GitHub itself got hacked

I noticed you didn't mention if 2FA was enabled for all admin accounts for this repo. This is a typical requirement in github organization accounts whose members work together as a group.

[2bndy5]

It is definitely an inconvenience though but here I feel like it's a trade-off worth having in this case.

@vermeeren So, I guess you disapprove of this PR's solution? Feel free to close it.

FWIW, I think the PGP signature (and indeterminate 2FA usage on this repo's admin accounts) is the main reason distributed uploads to pypi cannot be automated.

[svenevs]

I didn't think building the extension from a .git-less source is a huge priority.

This is something that's rather common, and also what pypa/build encourages. The package managers writ large also want these tarballs without git source. I don't really have any horses in this race, but this is why I never adopted setuptools_scm. It's otherwise beautiful. Adding some alternative considerations on the issue since currently I can't do any dev installs of breathe in CI.

[2bndy5]

currently I can't do any dev installs of breathe in CI.

yeah, using tox is a pain for pkgs that don't properly pip install. I had trouble running exhale tests on windows because of the lxml pkg trying to pip install without access to VS's cl.exe 😞.

[michaeljones]

Just trying to catch up a bit, sorry for not being more involved. My understanding from reading this is that this PR introduces setuptools_scm which relies on using git tags to generate the version number in a file in the file bundle. This is useful as we're currently having issues with stuff trying to access the version number as we currently store it in a file that also imports other things (I think that is my recollection anyway.)

However we're struggling to use the setuptools_scm approach because our typical release strategy runs the setup.py in a folder created from a git archive (tarball) and that doesn't include the git repository and so can't access the tags. It seems that doing releases from bare/non-git folders is relatively standard practice and so we can't adopt the setuptools_scm approach and we should close this PR and try to find another solution to the version number issue?

Is that a fair summary? Thanks for @2bndy5's work either way and to @vermeeren for checking it over. Our readthedocs builds are still failing in a manner related to this issue so I think it would be good to sort something out that I think I'd probably go for the less than ideal set up of just having the version number repeated in a couple of places with at least one of them easy to import by applications.

[michaeljones]

Our readthedocs build is failing with:

    File "/tmp/pip-build-env-oli4rdc4/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 268, in run_setup
      self).run_setup(setup_script=setup_script)
    File "/tmp/pip-build-env-oli4rdc4/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 158, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 11, in <module>
      from breathe import __version__
    File "/home/docs/checkouts/readthedocs.org/user_builds/breathe/checkouts/latest/breathe/__init__.py", line 1, in <module>
      from breathe.directives.setup import setup as directive_setup
    File "/home/docs/checkouts/readthedocs.org/user_builds/breathe/checkouts/latest/breathe/directives/__init__.py", line 1, in <module>
      from breathe.finder.factory import FinderFactory
    File "/home/docs/checkouts/readthedocs.org/user_builds/breathe/checkouts/latest/breathe/finder/__init__.py", line 1, in <module>
      from breathe.project import ProjectInfo
    File "/home/docs/checkouts/readthedocs.org/user_builds/breathe/checkouts/latest/breathe/project.py", line 3, in <module>
      from sphinx.application import Sphinx
  ModuleNotFoundError: No module named 'sphinx'

Which I think is due to setup.py attempt to import __version__ from breathe and getting the whole package with it which then fails due to sphinx not being available.

[2bndy5]

Yes. You completely understood the dialogue here.

@svenevs posted alternative options that would better suite the current release paradigm.

[michaeljones]

Thanks for checking. Sorry this has stretched out so long. Thanks again for your efforts.

[vermeeren]

A bit late, but I definitely liked this patchset, when I first saw it I thought it would really help with the simplification of version management. Just that upon closer look it turned out to cause an issue with the way packaging (afaik) has to be done.

So really thanks for all your efforts @2bndy5 it's really appreciated. (And of course, everyone else doing this for free in spare time!)

[JasperCraeghs]

I recommend the use of setuptools-scm to get the version from GIT even if it requires changes to the packaging strategy. My colleagues and I make use of it as much as possible.

Installing a modified version of breathe with pip install git+https://github.com/breathe-doc/breathe.git@my-branch does not override the latest version installed from PyPI because the version number is identical. As a workaround I could use pip's --force-reinstall --no-deps. I'm not supposed to edit the __version__ variable in my branch, am I?

Setuptools-scm adds a suffix to the version if there are changes compared to the GIT tag, e.g. 4.35.0.dev149+g5197d0f.d20230727.