GH Actions (docker): Run a job for "make build-local" first, cache image for job "make build"

[mkoeppe]

Follow up from #32703, where this was implemented for macOS.

Here we implement it for the docker-based workflows
via incremental builds (#34228) on top of previous stages pushed to ghcr.io

Depends on #34115
Depends on #34228

CC: @tobiasdiez @dimpase

Component: porting

Author: Matthias Koeppe

Branch/Commit: 5713937

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/33062

[mkoeppe]

Dependencies: #30933, #33103, #33140

[mkoeppe]

Changed dependencies from #30933, #33103, #33140 to #33103, #33140, #33222

[mkoeppe]

Changed dependencies from #33103, #33140, #33222 to #33103, #33140

[mkoeppe]

Branch: u/mkoeppe/gh_actions__docker___run_a_job_for__make_build_local__first__cache_image_for_job__make_build_

[mkoeppe]

Last 10 new commits:

531cd14	Merge #33103
51191fe	.github/workflows/tox.yml (local-macos): Fix filtering of experimental packages
12f88cf	.github/workflows/tox.yml (local-macos): Group optional/experimental package builds into fewer jobs
49236dd	tox.ini: Add local-macos-python3_pythonorg-python3.9 etc., refactor CONFIG_CONFIGURE_ARGS_1=...--with-python
1fcdc91	tox.ini: Updates for macos and manylinux from #31396
41ec3aa	tox.ini: Reindent
d7066aa	Merge #33140
98a5bae	Merge tag '9.6.beta0' into t/33062/gh_actions__docker___run_a_job_for__make_build_local__first__cache_image_for_job__make_build_
c446d33	.github/workflows, src/doc/en/developer/portability_testing.rst: Switch from docker.pkg.github.com to ghcr.io
c9c7d9f	tox.ini (docker): Support Docker build layer caching

[mkoeppe]

Commit: c9c7d9f

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

e56bf3e	.github/workflows/tox.yml: Integrate optional and experimental runs here
c830cdb	.github/workflows/ci-{linux,macos}.yml: Split from tox.yml

[sagetrac-git]

Changed commit from c9c7d9f to c830cdb

[mkoeppe]

Author: Matthias Koeppe

[mkoeppe]

Reviewer: https://github.com/mkoeppe/sage/actions/runs/1804137820

[mkoeppe]

comment:11

Obviously there is again a lot of cut+paste going on here, but I'm not sure yet whether this is best refactored using (1) scripts to be put in .ci/; (2) new top-level Makefile targets; (3) reusable workflows in the same repo; (4) reusable workflows from a different repo. So until then, copy+paste it is

[sagetrac-git]

Changed commit from c830cdb to 47b636d

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

47b636d

.github/workflows/ci-linux.yml: Fix typo

[tobiasdiez]

comment:14

A lot of jobs are failing because there doesn't exist a corresponding maximal image.

What is the distinction between the docker and docker-optional jobs?

[mkoeppe]

comment:15

Replying to @tobiasdiez:

What is the distinction between the docker and docker-optional jobs?

docker-optional also runs the stages that build optional and experimental packages.

[mkoeppe]

comment:17

Replying to @tobiasdiez:

A lot of jobs are failing because there doesn't exist a corresponding maximal image.

Not sure if that's it but there are a number of new failures introduced by switching to DOCKER_BUILDKIT=1, for example https://github.com/mkoeppe/sage/runs/5087229478?check_suite_focus=true

 > [with-system-packages 130/204] RUN zypper --ignore-unknown install --no-confirm --auto-agree-with-licenses --no-recommends --details "pkgconfig(gf2x)" || echo "(ignoring error)":
------
failed to prepare d4bec9ys962tp15dnd39jkkya: max depth exceeded

This will require some changes in build/bin/write-dockerfile.sh

[tobiasdiez]

comment:18

Replying to @mkoeppe:

Replying to @tobiasdiez:

What is the distinction between the docker and docker-optional jobs?

docker-optional also runs the stages that build optional and experimental packages.

Okay, thanks! Would it then make sense to base the docker-optional on the image created in the docker job to speed up the build (i.e. as a "stage")?

[mkoeppe]

comment:19

It's currently using a different package configuration (-maximal instead of -standard)

[mkoeppe]

Work Issues: TARGETS_OPTIONAL not passed correctly

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

5713937

sage -package list --has-file, --no-file: Handle disjunctions

[sagetrac-git]

Changed commit from ca6d163 to 5713937

[mkoeppe]

Changed work issues from TARGETS_OPTIONAL not passed correctly to none

[mkoeppe]

comment:85

Marking it as critical because it is a dependency of #34266

[dimpase]

comment:87

lgtm

[dimpase]

Changed reviewer from https://github.com/mkoeppe/sage/actions/runs/2751836720 to Dima Pasechnik

[mkoeppe]

comment:88

Thanks!

[vbraun]

Changed branch from u/mkoeppe/gh_actions__docker___run_a_job_for__make_build_local__first__cache_image_for_job__make_build_ to 5713937