sagelib: Generate pyproject.toml and setup.cfg [install_requires], requirements.txt, Pipfile, and src/Pipfile

[mkoeppe]

After #29500, it is possible to install Python packages built by the Sage distribution into user-defined venvs.

As of Sage 9.3.beta8, sagelib is published as a source distribution on PyPI (https://pypi.org/project/sagemath-standard), but the package does not have proper dependency information yet.

As part of bootstrap, we now generate (1) the pyproject.toml (PEP 517 metadata) and install_requires information, (2) requirements.txt and (3) Pipfile using m4 from the information added in #30719. These files provide 3 distinct ways of setting up a user-defined venv.

(Adding pyproject.toml does not change how the Sage distribution installs sagelib because build/pkgs/sagelib/spkg-install uses setup.py install directly.)

To test - use ./bootstrap and then configure and build the Sage distribution. Then follow the instructions in build/pkgs/sagelib/src/tox.ini:

    #
    # SUPPORTED ENVIRONMENTS:
    #
    # Build dependencies according to requirements.txt (all versions fixed).
    # Use ONLY the wheels built and stored by the Sage distribution (no PyPI):
    #
    #   ./sage -sh -c '(cd build/pkgs/sagelib/src && tox -v -v -v -e python-sagewheels-nopypi)'
    #
    python-sagewheels-nopypi,
    #
    # Build and test without using the concrete dependencies specified by requirements.txt,
    # using the dependencies declared in pyproject.toml and setup.cfg (install-requires) only:
    # Still use ONLY the wheels built and stored by the Sage distribution (no PyPI).
    #
    #   ./sage -sh -c '(cd build/pkgs/sagelib/src && tox -v -v -v -e python-sagewheels-nopypi-norequirements)'
    #
    python-sagewheels-nopypi-norequirements,
    #

Related or follow-up tickets:

• ./configure --enable-editable #31377 ./configure --enable-editable
• Developer's Guide: Add instructions for sagelib development in a conda environment #28746 Developer's Guide: Add instructions for sagelib development in a conda environment
• Meta-ticket: Make sagelib pip-installable in a conda environment #28752 Make sagelib pip-installable in a conda environment
• Use pipenv to create venv for development #30371 In-place (editable) installs of sagelib in a venv
• sage_bootstrap: Add command "sage -package list --output={install-requires,requirements,pipfile,debian,....}" #29041 sage_bootstrap: Add command "sage -package list --output={install-requires,requirements,pipfile,debian,....}"
• add documentation on how to set up the virtual environment, using either plain python -m venv, virtualenv (20.x), or pipenv.

References:

• https://snarky.ca/what-the-heck-is-pyproject-toml/

CC: @dimpase @tobiasdiez @jhpalmieri @videlec @kiwifb @malb

Component: build

Keywords: sd111

Author: Matthias Koeppe

Branch/Commit: 04da2c6

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/30913

[mkoeppe]

Changed dependencies from #30578, #30912 to #30578, #30912, #30719

[mkoeppe]

Branch: u/mkoeppe/sagelib__add_setup_cfg__install_requires_

[mkoeppe]

Author: Matthias Koeppe

[mkoeppe]

Commit: b0f5cd2

[mkoeppe]

Last 10 new commits:

e61929d	build/pkgs/sphinx/install-requires.txt: Update from gentoo
8fe0e35	build/pkgs/python_openid: Unused, remove
ba1d913	build/pkgs/itsdangerous: Unused, remove
bb6c4ae	build/pkgs/tox/install-requires.txt: New
d507501	build/pkgs/networkx/install-requires.txt: Update from gentoo
c3a9352	build/pkgs/pathpy/install-requires.txt: Package removed in #30611, remove
ebd4610	build/pkgs/pynac/install-requires.txt: New
6ec00dd	build/pkgs/{numpy,pillow}/install-requires.txt: New
6bff345	Merge branch 't/30719/add_build_pkgs_spkg_install_requires_txt_for_all_python_packages' into t/30913/sagelib__add_setup_cfg__install_requires_
b0f5cd2	bootstrap: Generate build/pkgs/sagelib/src/setup.cfg with m4

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

6094c68

build/pkgs/cypari/install-requires.txt: cypari2, not cypari

[sagetrac-git]

Changed commit from b0f5cd2 to 6094c68

[sagetrac-git]

Changed commit from 6094c68 to 411cbfb

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

411cbfb

build/bin/sage-get-system-packages: Add comment

[sagetrac-git]

Changed commit from 411cbfb to 3b02c3d

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

3b02c3d

bootstrap: Run bootstrap scripts in build/pkgs/*/, create build/pkgs/sagelib/bootstrap

[kiwifb]

comment:10

Will this be able to deal with optional package declarations (in time)?

[mkoeppe]

comment:11

The plan for optional packages is that users would install a package such as sagemath-meataxe or sagemath-tdlib in addition to sagemath-standard.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

95b74e8

Merge tag '9.3.beta2' into t/30913/sagelib__add_setup_cfg__install_requires_

[sagetrac-git]

Changed commit from 3b02c3d to 95b74e8

[mkoeppe]

Changed keywords from none to sd111

[mkoeppe]

Changed dependencies from #30578, #30912, #30719 to #30578, #30912

[sagetrac-git]

Changed commit from 95b74e8 to 8b8b9f9

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

99b56f5	build/pkgs/sagelib/src/tox.ini: Add environment python-norequirements
8b8b9f9	Merge tag '9.3.beta3' into t/30913/sagelib__add_setup_cfg__install_requires_

[mkoeppe]

comment:16

Added instructions for testing to the ticket description.
In particular, this reveals the missing dependencies of upstream Python packages tracked in #30922.

[sagetrac-git]

Changed commit from 8b8b9f9 to 3382a47

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

3382a47

build/pkgs/pplpy/install-requires.txt: Switch to git+https link

[sagetrac-git]

Changed commit from 3382a47 to bf61db4

[mkoeppe]

comment:148

pip likes to use temporary directories, probably in /tmp or /var/tmp

[mkoeppe]

comment:149

Try setting TMPDIR to something elese

[dimpase]

comment:150

is TMPDIR an env.var. I can export before running this job? Or is it something pythonic?

[mkoeppe]

comment:151

Just the standard environment variable

[dimpase]

comment:152

OK, this run finishes, with some doctest errors, mainly in sphinx/docs related:

$ ./sage -sh -c '(cd build/pkgs/sagelib/src && tox -v -v -v -e python-sagewheels-nopypi)'
...
sage -t --random-seed=0 /home/scratch2/dimpase/sage/sage/src/sage/all.py  # 1 doctest failed
sage -t --random-seed=0 /home/scratch2/dimpase/sage/sage/src/sage/misc/package.py  # 1 doctest failed
sage -t --random-seed=0 /home/scratch2/dimpase/sage/sage/src/sage/misc/sagedoc.py  # 4 doctests failed
sage -t --random-seed=0 /home/scratch2/dimpase/sage/sage/src/sage/docs/conf.py  # 1 doctest failed
sage -t --random-seed=0 /home/scratch2/dimpase/sage/sage/src/sage/interfaces/gap.py  # 1 doctest failed
sage -t --random-seed=0 /home/scratch2/dimpase/sage/sage/src/sage_setup/find.py  # 1 doctest failed
sage -t --random-seed=0 /home/scratch2/dimpase/sage/sage/src/sage_docbuild/__init__.py  # 35 doctests failed
sage -t --random-seed=0 /home/scratch2/dimpase/sage/sage/src/sage_docbuild/sphinxbuild.py  # 13 doctests failed
sage -t --random-seed=0 /home/scratch2/dimpase/sage/sage/src/sage_docbuild/utils.py  # 5 doctests failed

if this is expected, feel free to make it positive review.

[dimpase]

Reviewer: Dima Pasechnik

[mkoeppe]

comment:153

Thanks for testing! Yes, this looks as expected. The documentation is not built by this invocation. We should disable the corresponding doctests, for example using #30778 - but that's something for the 9.4 series.

[mkoeppe]

comment:154

Setting priority to blocker to bring this ticket to the attention of the release bot.

[vbraun]

Changed branch from u/mkoeppe/sagelib__add_setup_cfg__install_requires_ to 04da2c6