tox.ini: Add environments local-sudo-ubuntu-standard, etc.

[mkoeppe]

These environments would just run sudo apt-get ... etc. to install (but not remove!) packages, for use at the user's own risk.

To test (on an Ubuntu system):

    tox -e local-sudo-ubuntu-bionic-standard

or

    tox -e local-sudo-standard

CC: @tobiasdiez @dimpase

Component: porting

Author: Matthias Koeppe

Branch: ff34897

Reviewer: Tobias Diez, Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/30923

[mkoeppe]

Author: Matthias Koeppe

[tobiasdiez]

comment:2

Is it possible to put the sudo outside? I.e. sudo tox -r local-ubuntu-standard?

[mkoeppe]

comment:3

But we would only want to do the package installation as root, not the Sage build

[mkoeppe]

Branch: u/mkoeppe/tox_ini__add_environments_local_sudo_ubuntu_standard__etc_

[mkoeppe]

Commit: 30e624a

[mkoeppe]

comment:5

Here's a first approximation.

---

New commits:

30e624a

tox.ini: Add local-sudo

[tobiasdiez]

comment:7

I think you forgot the confirmation -y after install: tobiasdiez@942149e

Moreover, I was wondering if one should also read the debian-bootstrap.txt files.

Finally, there are problems that some of the packages are not found. First, one should probably add sudo add-apt-repository ppa:deadsnakes/ppa since otherwise libpython3.7-dev is not found on ubuntu 20.04. Second, texlive-generic-extra doesn't exist on ubuntu 20.04.
https://github.com/tobiasdiez/sage/runs/1404001507?check_suite_focus=true

Other than that, looks good to me! Thanks a lot.

[mkoeppe]

comment:8

Yes, I'll add bootstrap

I don't think we have libpython3.7-dev listed in the debian.txt files any more.

But yes, handling IGNORE_MISSING_SYSTEM_PACKAGES needs to be done.

For deadsnakes, there's #30217. This would not be the default, but a mix-in flavor.

[tobiasdiez]

comment:9

On a second thought, do you really thing it's the job of tox to install system packages? I would feel more comfortable to put the code into a shell script that you can run as build/install_system_packages ubuntu-2004, say. (Having such a script might be a good idea anyway).

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

75ecd11

tox.ini (local-sudo): Also use ...-bootstrap.txt

[sagetrac-git]

Changed commit from 30e624a to 75ecd11

[tobiasdiez]

comment:11

Replying to @mkoeppe:

But yes, handling IGNORE_MISSING_SYSTEM_PACKAGES needs to be done.

and/or change texlive-generic-extra that exist on modern ubuntu systems as well.

[tobiasdiez]

comment:12

Replying to @sagetrac-git:

Branch pushed to git repo; I updated commit sha1. New commits:

75ecd11

tox.ini (local-sudo): Also use ...-bootstrap.txt

Don't forget to change local-sudo-standard/maximal as well.

[mkoeppe]

comment:13

Replying to @tobiasdiez:

On a second thought, do you really thing it's the job of tox to install system packages? I would feel more comfortable to put the code into a shell script that you can run as build/install_system_packages ubuntu-2004, say. (Having such a script might be a good idea anyway).

It's working well with tox, in particular the environment factors give a nice compact way of expressing different configurations.

And I definitely do not want to add user-facing scripts to Sage that promises to install all necessary system packages. We deliberately only offer system package advice (at the end of ./configure). Already now users complain if that advice is not entirely accurate -- so we definitely do not want to make stronger promises! We simply do not have enough developers who could keep it up to date.

But there is a lot of room for refactoring what is in tox.ini into various helper scripts. I have described some possible directions in #30865 -- help is very welcome there.

[mkoeppe]

comment:14

Replying to @tobiasdiez:

Replying to @sagetrac-git:

Branch pushed to git repo; I updated commit sha1. New commits:

75ecd11	tox.ini (local-sudo): Also use ...-bootstrap.txt

Don't forget to change local-sudo-standard/maximal as well.

No, that line is also executed for these environments - by the magic of tox's factor conditions.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

0dc79eb	build/bin/sage-print-system-package-command: Handle --no-install-recommends, --yes for systems for which write-dockerfile.sh knows these flags
050dcb8	tox.ini (local-sudo): Use --yes --no-install-recommends

[sagetrac-git]

Changed commit from 75ecd11 to 050dcb8

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

c8fbe0b

tox.ini (local-sudo): Ignore errors when IGNORE_MISSING_SYSTEM_PACKAGES=yes

[sagetrac-git]

Changed commit from 050dcb8 to c8fbe0b

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

ff34897

tox.ini (local): Guess the package system if it is not provided as a factor

[sagetrac-git]

Changed commit from c8fbe0b to ff34897

[tobiasdiez]

comment:20

Replying to @mkoeppe:

Replying to @tobiasdiez:

On a second thought, do you really thing it's the job of tox to install system packages? I would feel more comfortable to put the code into a shell script that you can run as build/install_system_packages ubuntu-2004, say. (Having such a script might be a good idea anyway).

It's working well with tox, in particular the environment factors give a nice compact way of expressing different configurations.

Sure, but it feels a bit like you are using tox for something it wasn't designed. Normally, in a tox setting, "environment" refers to a virtual python env, but for sage it is the complete system env including the os and system packages. I have the feeling a more general-purpose framework like invoke would be a better solution to manage these things.

And I definitely do not want to add user-facing scripts to Sage that promises to install all necessary system packages.

I didn't meant it as a user-facing script, but only for developers (i.e don't ship it).

[tobiasdiez]

comment:21

I've tested the current code in the wsl github workflow and it seems to work well: https://github.com/tobiasdiez/sage/actions/runs/365823571

[mkoeppe]

comment:22

Sounds like a positive review?

[tobiasdiez]

Reviewer: Tobias Diez, ...

[tobiasdiez]

comment:23

Yes, in principle. But I cannot really judge the changes in the bash script.

[dimpase]

comment:25

ok

[dimpase]

Changed reviewer from Tobias Diez, ... to Tobias Diez, Dima Pasechnik

[mkoeppe]

comment:26

Thanks!

[tobiasdiez]

comment:27

Is it possible to call only the system package install code via tox (i.e. don't run rest of build and tests)?

That would be handy for #30371.

[mkoeppe]

comment:28

Replying to @tobiasdiez:

Is it possible to call only the system package install code via tox (i.e. don't run rest of build and tests)?

Yes, you can pass make targets to tox, e.g., tox -e local-sudo-standard -- configure will only run bootstrap, or tox -e local-sudo-standard -- config.status will stop after running ./configure

[tobiasdiez]

comment:29

Thanks! That kind of worked. I had to install sudo manually, since this is not installed by default on github actions and also not necessary.
It would be nice if you could try to make it work without sudo.

local-sudo-standard run-test: commands[2] | bash -c 'PACKAGES=`sed "s/#.*//;" build/pkgs/$(build/bin/sage-guess-package-system)*.txt`; $(build/bin/sage-print-system-package-command $(build/bin/sage-guess-package-system) --sudo --yes --no-install-recommends install $PACKAGES) || [ "$IGNORE_MISSING_SYSTEM_PACKAGES" = yes ] && echo "(ignoring errors)" '
/usr/bin/bash: sudo: command not found

ERROR: InvocationError for command /usr/bin/bash -c 'PACKAGES=`sed "s/#.*//;" build/pkgs/$(build/bin/sage-guess-package-system)*.txt`; $(build/bin/sage-print-system-package-command $(build/bin/sage-guess-package-system) --sudo --yes --no-install-recommends install $PACKAGES) || [ "$IGNORE_MISSING_SYSTEM_PACKAGES" = yes ] && echo "(ignoring errors)" ' (exited with code 1)

Edit: Moreover, DEBIAN_FRONTEND: noninteractive was/is missing. And it fails with

configure: error: You cannot build Sage as root, switch to an unprivileged user.  (If building in a container, use --enable-build-as-root.)

Finally, tox -e local-sudo-standard -- config.status doesn't stop after contigure and continues with make, see https://github.com/tobiasdiez/sage/runs/1438029552?check_suite_focus=true

Since using this tox command for ci was one of the motivations but doesn't really work at the moment without extra modifications, I'll set the ticket back to needs work.

[vbraun]

Changed branch from u/mkoeppe/tox_ini__add_environments_local_sudo_ubuntu_standard__etc_ to ff34897

[vbraun]

comment:32

Can you make a separate ticket for that? Once a ticket is in positive review I'll only unmerge it for critical bugs, not things that would also be good to have.

[vbraun]

Changed commit from ff34897 to none

[tobiasdiez]

comment:33

Sure! It's now #30944. Sorry if I broke any (unwritten) laws; I wasn't sure about resetting it to "needs work".