GH Actions (macOS): Run a job for "make build-local" first, cache image for job "make build"

[mkoeppe]

We revise the GH Actions workflows to use a 2-stage build:
In the first stage, run make build-local, and store SAGE_LOCAL as a build artifact. In the second stage, download the build artifact and run more building and testing.

(+) On top of the artifact containing the full SAGE_LOCAL, we can test several ways to build the Python parts

• Sage distribution, classic
• Sage distribution with configure --enable-editable
• Sage distribution with configure --enable-system-site-packages (configure --enable-system-site-packages via spkg-configure.m4 for cvxopt and other python packages #29665)
• the configurations from pkgs/sagemath-standard/tox.ini
• build/install of modularized distributions such as pkgs/sage-categories (Modularization of sagelib: Break out separate packages sagemath-objects, sagemath-categories #29865), pkgs/sagemath-standard-no-symbolics (Modularization of sagelib: Break out a separate package sagemath-standard-no-symbolics #32601)

(+) Tests of optional and experimental packages can be streamlined, as we avoid rebuilding their dependencies that are standard packages.

(+) Splitting the job into two would also help with the configurations for which we scrape at the 6 hour time limit

(-) Unfortunately, because because "needs" cannot depend on "matrix", the jobs for building/testing Python packages would not start before all jobs building SAGE_LOCAL for all platforms are completed

In this ticket, we only change all existing macos workflows to a 2-stage workflow, integrating the separate workflows for optional and experimental packages.

As of this ticket, we rely on the bottleneck of the available parallel jobs on GH Actions to ensure that the 2nd stages of a configuration are run after the 1st stage of that configuration. Experience with this workflow will show whether this suffices.

We also update the macOS/Xcode versions according to what's available on GH Actions and switch the homebrew builds to faster homebrew-usrlocal variants, which can use bottles for all available packages.

Depends on #32113
Depends on #32947

CC: @tobiasdiez @kliem @orlitzky @isuruf

Component: porting

Author: Matthias Koeppe

Branch/Commit: edb4364

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/32703

[mkoeppe]

Dependencies: #31535

[mkoeppe]

Branch: u/mkoeppe/gh_actions__run_a_job_for__make_build_local__first__cache_image_for_job__make_build_

[mkoeppe]

Reviewer: https://github.com/mkoeppe/sage/actions/runs/1350377940

[mkoeppe]

Commit: d08452c

[mkoeppe]

Last 10 new commits:

2101b8c	build/pkgs/python3/spkg-build.in: Make sure that python finds sqlite3 when determining which extension modules to build
3bbc5d8	build/pkgs/python3/spkg-build.in: Set rpath
124b605	Merge #32698
bca2141	pkgs/sagemath-standard/tox.ini: Use SAGE_VENV or venv symlink to find wheels
bdbab3a	build/make/Makefile.in: Remove SPKG-tox, SPKG-sdist targets
63e47ff	Merge #31535
1c40d1c	.github/workflows/extract-sage-local.sh: Make script usable on non-cygwin
dee3f41	github/workflows/tox.yml: Multi-stage local-macos
bd9fad1	.github/workflows/: Remove workflows for testing
d08452c	Update systems

[mkoeppe]

Changed reviewer from https://github.com/mkoeppe/sage/actions/runs/1350377940 to https://github.com/mkoeppe/sage/actions/runs/1350710395

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

0799c5f

.github/workflows/tox.yml: Fixup for macOS tar

[sagetrac-git]

Changed commit from d08452c to 0799c5f

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

e031622

.github/workflows/tox.yml: Fix up upload path

[sagetrac-git]

Changed commit from 0799c5f to e031622

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

a40d706

.github/workflows/tox.yml: Do not use /tmp for artifacts to avoid https://github.com/actions/upload-artifact/issues/92

[sagetrac-git]

Changed commit from e031622 to a40d706

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

dd2bc15	.github/workflows/tox.yml: Use env.GITHUB_WORKSPACE
2b46f02	.github/workflows/tox.yml: Use sage-local-artifact/ as path

[sagetrac-git]

Changed commit from a40d706 to 2b46f02

[mkoeppe]

Changed reviewer from https://github.com/mkoeppe/sage/actions/runs/1350710395 to https://github.com/mkoeppe/sage/runs/3935018641?check_suite_focus=true

[mkoeppe]

comment:11

working well now

[sagetrac-git]

Changed commit from 2b46f02 to 4dcd768

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

4dcd768

.github/workflows/tox.yml: mkdir

[sagetrac-git]

Changed commit from 4dcd768 to 828463d

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

828463d

github/workflows/tox.yml: Multi-stage local-macos

[tobiasdiez]

comment:14

If I understand the changes (and the output) correctly, then currently stage 1 and 2 run parallel. For example, some stage 2 runs (e.g https://github.com/mkoeppe/sage/runs/3935018795?check_suite_focus=true) fail since they try to download a non-existing artifact. Maybe something like https://github.com/marketplace/actions/wait-for-check helps.

[mkoeppe]

comment:15

Replying to @tobiasdiez:

If I understand the changes (and the output) correctly, then currently stage 1 and 2 run parallel.

Yes, in theory. But in practice we only get 5 parallel jobs on macOS and we launch many more jobs...

For example, some stage 2 runs (e.g https://github.com/mkoeppe/sage/runs/3935018795?check_suite_focus=true) fail since they try to download a non-existing artifact.

Well, they failed because the previous stage failed.

[mkoeppe]

comment:16

Replying to @tobiasdiez:

Maybe something like https://github.com/marketplace/actions/wait-for-check helps.

Thanks for the pointer, we can look into using something like this

[mkoeppe]

comment:17

Testing with integrated builds for the optional packages now at https://github.com/mkoeppe/sage/actions/runs/1360303527

[tobiasdiez]

comment:18

Replying to @mkoeppe:

Replying to @tobiasdiez:

If I understand the changes (and the output) correctly, then currently stage 1 and 2 run parallel.

Yes, in theory. But in practice we only get 5 parallel jobs on macOS and we launch many more jobs...

Okay, but isn't this a very fragile design? Imagine that you have exactly 5 matrix cases. So all of the stage 1 tasks start at the same time. Now if the last case exists slightly before the other 4, then the stage 2 for the other 4 may be invoked and now fail as their stage 1 is not yet finished. Also this would limit everything to mac, as on linux you have way more runners.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

43d481b	bump to 0.1.0
f66821b	cysignals are needed
ae37b4e	deprecate sage.interfaces.primecount, not just remove
d3ae5f4	primecount is on conda, too
3a8c7fe	primesieve is on conda, too
74b3845	allow float inputs for prime_pi
591be34	Merge #32894
049a5f6	tox.ini: Do not set environment variable CONDARC
f3116a1	tox.ini (conda): Force use of conda's python3
6418579	Merge #32113

[sagetrac-git]

Changed commit from 6418579 to 933134c

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

933134c

build/pkgs/gfortran/distros/homebrew.txt: Use gfortran

[mkoeppe]

Changed reviewer from https://github.com/mkoeppe/sage/actions/runs/1597453181 to https://github.com/mkoeppe/sage/actions/runs/1605145773

[mkoeppe]

Changed dependencies from #32113 to #32113, #32947

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

ab4fb99	tox.ini: Add ubuntu-jammy, debian-bookworm, linuxmint-20.3, fedora-36
e51225a	.github/workflows/tox*.yml: Remove debian-jessie, add ubuntu-jammy, debian-bookworm, linuxmint-20.3, fedora-36
ac47e7d	.github/workflows/tox-gcc_spkg.yml: Remove
1aa9328	Merge #32947
e99a09e	sed -i.bak 's/ubunty/ubuntu/g' .github/workflows/*.yml

[sagetrac-git]

Changed commit from 933134c to e99a09e

[tobiasdiez]

comment:48

I know we already touch upon this point, but can you please expand on

Unfortunately, because ​because "needs" cannot depend on "matrix", the jobs for building/testing Python packages would not start before all jobs building SAGE_LOCAL for all platforms are completed

That is, why do you prefer the "custom stage build" over having a "local-macos-stage2" job that depends via "needs" on "local-macos-stage1", where each of these jobs defines an appropriate matrix. In particular, I don't understand the "all platforms" part, as ubuntu and macos are in different jobs, right? It appears to me that with the current design also stage 1 builds are executed before any stage 2 is executed, so the only difference is that you can already have 4 or so stage 2 builds running while the last stage 1 build finishes. I guess for the overall build time this shouldn't make much of a difference anyway, as these 4 runners would be busy with other workflows then (or are free to execute builds of other stuff in the sagemath org).

[mkoeppe]

comment:49

Replying to @tobiasdiez:

That is, why do you prefer the "custom stage build" over having a "local-macos-stage2" job that depends via "needs" on "local-macos-stage1", where each of these jobs defines an appropriate matrix.

To avoid copy-paste

[mkoeppe]

comment:50

Replying to @tobiasdiez:

I don't understand the "all platforms" part, as ubuntu and macos are in different jobs, right?

Yes, that's right, within each job; and in this ticket, only for macos.

[tobiasdiez]

comment:51

Replying to @mkoeppe:

Replying to @tobiasdiez:

That is, why do you prefer the "custom stage build" over having a "local-macos-stage2" job that depends via "needs" on "local-macos-stage1", where each of these jobs defines an appropriate matrix.

To avoid copy-paste

Github recently made it possible to reuse workflows: https://docs.github.com/en/actions/learn-github-actions/reusing-workflows
Thus you could extract the current local-macos job into a new workflow and pass the right "tox" command as an argument (determined by the matrix + step).

[mkoeppe]

comment:52

Thanks for the pointer!! I've added this to #29060 for welcome future refactoring - but this won't make it into Sage 9.5

[sagetrac-git]

Changed commit from e99a09e to edb4364

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

edb4364

.github/workflows/tox.yml: Replace homebrew-macos-python3_xcode-standard by homebrew-macos-usrlocal-python3_xcode-standard

[dimpase]

Changed reviewer from https://github.com/mkoeppe/sage/actions/runs/1605145773 to Dima Pasechnik

[dimpase]

comment:55

OK,macOS runs on GH look good.

[mkoeppe]

comment:56

Thanks!

[vbraun]

Changed branch from u/mkoeppe/gh_actions__run_a_job_for__make_build_local__first__cache_image_for_job__make_build_ to edb4364