feat!: Rewrite build system and third-party dependencies (shaka-proje…

…ct#1310)

This work was done over ~80 individual commits in the `cmake` branch,
which are now being merged back into `main`. As a roll-up commit, it is
too big to be reviewable, but each change was reviewed individually in
context of the `cmake` branch. After this, the `cmake` branch will be
renamed `cmake-porting-history` and preserved.

---------

Co-authored-by: Geoff Jukes <[email protected]>
Co-authored-by: Bartek Zdanowski <[email protected]>
Co-authored-by: Carlos Bentzen <[email protected]>
Co-authored-by: Dennis E. Mungai <[email protected]>
Co-authored-by: Cosmin Stejerean <[email protected]>
Co-authored-by: Carlos Bentzen <[email protected]>
Co-authored-by: Cosmin Stejerean <[email protected]>
Co-authored-by: Cosmin Stejerean <[email protected]>

.github/workflows/README.md

@@ -1,39 +1,63 @@
 # GitHub Actions CI
 
-## Actions
- - `custom-actions/lint-packager`:
-   Lints Shaka Packager.  You must pass `fetch-depth: 2` to `actions/checkout`
-   in order to provide enough history for the linter to tell which files have
-   changed.
- - `custom-actions/build-packager`:
-   Builds Shaka Packager.  Leaves build artifacts in the "artifacts" folder.
-   Requires OS-dependent and build-dependent inputs.
- - `custom-actions/test-packager`:
-   Tests Shaka Packager.  Requires OS-dependent and build-dependent inputs.
- - `custom-actions/build-docs`:
-   Builds Shaka Packager docs.
-
-## Workflows
- - On PR:
-   - `build_and_test.yaml`:
-     Builds and tests all combinations of OS & build settings.  Also builds
-     docs.
- - On release tag:
-   - `github_release.yaml`:
-     Creates a draft release on GitHub, builds and tests all combinations of OS
-     & build settings, builds docs on all OSes, attaches static release binaries
-     to the draft release, then fully publishes the release.
- - On release published:
-   - `docker_hub_release.yaml`:
-     Builds a Docker image to match the published GitHub release, then pushes it
-     to Docker Hub.
-   - `npm_release.yaml`:
-     Builds an NPM package to match the published GitHub release, then pushes it
-     to NPM.
-   - `update_docs.yaml`:
-     Builds updated docs and pushes them to the gh-pages branch.
+## Reusable workflows
+ - `build.yaml`:
+   Build and test all combinations of OS & build settings.  Also builds docs on
+   Linux.
+
+ - `build-docs.yaml`:
+   Build Packager docs.  Runs only on Linux.
+
+ - `build-docker.yaml`:
+   Build the official Docker image.
+
+ - `lint.yaml`:
+   Lint Shaka Packager.
+
+ - `publish-docs.yaml`:
+   Publish Packager docs.  Runs on the latest release.
+
+ - `publish-docker.yaml`:
+   Publish the official docker image.  Runs on all releases.
+
+ - `publish-npm.yaml`:
+   Publish binaries to NPM.  Runs on all releases.
+
+ - `test-linux-distros.yaml`:
+   Test the build on all Linux distros via docker.
+
+## Composed workflows
+ - On PR (`pr.yaml`), invoke:
+   - `lint.yaml`
+   - `build.yaml`
+   - `build-docs.yaml`
+   - `build-docker.yaml`
+   - `test-linux-distros.yaml`
+
+## Release workflow
+ - `release-please.yaml`
+   - Updates changelogs, version numbers based on conventional commits syntax
+     and semantic versioning
+     - https://conventionalcommits.org/
+     - https://semver.org/
+   - Generates/updates a PR on each push
+   - When the PR is merged, runs additional steps:
+     - Creates a GitHub release
+     - Invokes `publish-docs.yaml` to publish the docs
+     - Invokes `publish-docker.yaml` to publish the docker image
+     - Invokes `build.yaml`
+     - Attaches the binaries from `build.yaml` to the GitHub release
+     - Invokes `publish-npm.yaml` to publish the binaries to NPM
+
+## Common workflows from shaka-project
+ - `sync-labels.yaml`
+ - `update-issues.yaml`
+ - `validate-pr-title.yaml`
 
 ## Required Repo Secrets
+ - `RELEASE_PLEASE_TOKEN`: A PAT for `shaka-bot` to run the `release-please`
+   action.  If missing, the release workflow will use the default
+   `GITHUB_TOKEN`
  - `DOCKERHUB_CI_USERNAME`: The username of the Docker Hub CI account
  - `DOCKERHUB_CI_TOKEN`: An access token for Docker Hub
    - To generate, visit https://hub.docker.com/settings/security
@@ -47,3 +71,12 @@
  - `NPM_PACKAGE_NAME`: Not a true "secret", but stored here to avoid someone
    pushing bogus packages to NPM during CI testing from a fork
    - In a fork, set to a private name which differs from the production one
+
+## Repo Settings
+
+Each of these workflow features can be enabled by creating a "GitHub
+Environment" with the same name in your repo settings.  Forks will not have
+these enabled by default.
+
+ - `debug`: enable debugging via SSH after a failure
+ - `self_hosted`: enable self-hosted runners in the build matrix

.github/workflows/build-docker.yaml

@@ -0,0 +1,37 @@
+# Copyright 2022 Google LLC
+#
+# Use of this source code is governed by a BSD-style
+# license that can be found in the LICENSE file or at
+# https://developers.google.com/open-source/licenses/bsd
+
+# A workflow to build the official docker image.
+name: Official Docker image
+
+# Runs when called from another workflow.
+on:
+  workflow_call:
+    inputs:
+      ref:
+        required: true
+        type: string
+
+# By default, run all commands in a bash shell.  On Windows, the default would
+# otherwise be powershell.
+defaults:
+  run:
+    shell: bash
+
+jobs:
+  official_docker_image:
+    name: Build
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v3
+        with:
+          ref: ${{ inputs.ref }}
+          submodules: recursive
+
+      - name: Build
+        shell: bash
+        run: docker buildx build .

.github/workflows/build-docs.yaml

@@ -0,0 +1,77 @@
+# Copyright 2022 Google LLC
+#
+# Use of this source code is governed by a BSD-style
+# license that can be found in the LICENSE file or at
+# https://developers.google.com/open-source/licenses/bsd
+
+# A reusable workflow to build Packager docs.  Leaves docs output in the
+# "gh-pages" folder.  Only runs in Linux due to the dependency on doxygen,
+# which we install with apt.
+name: Build Docs
+
+# Runs when called from another workflow.
+on:
+  workflow_call:
+    inputs:
+      ref:
+        required: true
+        type: string
+
+      # If true, start a debug SSH server on failures.
+      debug:
+        required: false
+        type: boolean
+        default: false
+
+jobs:
+  docs:
+    name: Build docs
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Install dependencies
+        run: |
+          sudo apt install -y doxygen
+          python3 -m pip install \
+              sphinx==7.1.2 \
+              sphinxcontrib.plantuml \
+              recommonmark \
+              cloud_sptheme \
+              breathe
+
+      - name: Checkout code
+        uses: actions/checkout@v3
+        with:
+          ref: ${{ inputs.ref }}
+
+      - name: Generate docs
+        run: |
+          mkdir -p gh-pages
+          mkdir -p build
+
+          # Doxygen must run before Sphinx.  Sphinx will refer to
+          # Doxygen-generated output when it builds its own docs.
+          doxygen docs/Doxyfile
+
+          # Now build the Sphinx-based docs.
+          make -C docs/ html
+
+          # Now move the generated outputs.
+          cp -a build/sphinx/html gh-pages/html
+          cp -a build/doxygen/html gh-pages/docs
+          cp docs/index.html gh-pages/index.html
+
+          # Now set permissions on the generated docs.
+          # https://github.com/actions/upload-pages-artifact#file-permissions
+          chmod -R +rX gh-pages/
+
+      - name: Upload docs artifacts
+        uses: actions/upload-pages-artifact@v2
+        with:
+          path: gh-pages
+
+      - name: Debug
+        uses: mxschmitt/[email protected]
+        with:
+          limit-access-to-actor: true
+        if: failure() && inputs.debug

.github/workflows/build-matrix.json

@@ -0,0 +1,38 @@
+{
+  "comment1": "runners hosted by GitHub, always enabled",
+  "hosted": [
+    {
+      "os": "ubuntu-latest",
+      "os_name": "linux",
+      "target_arch": "x64",
+      "exe_ext": "",
+      "generator": "Ninja"
+    },
+    {
+      "os": "macos-latest",
+      "os_name": "osx",
+      "target_arch": "x64",
+      "exe_ext": "",
+      "generator": "Ninja"
+    },
+    {
+      "os": "windows-latest",
+      "os_name": "win",
+      "target_arch": "x64",
+      "exe_ext": ".exe",
+      "generator": ""
+    }
+  ],
+
+  "comment2": "runners hosted by the owner, enabled by the 'self_hosted' environment being created on the repo",
+  "selfHosted": [
+    {
+      "os": "self-hosted-linux-arm64",
+      "os_name": "linux",
+      "target_arch": "arm64",
+      "exe_ext": "",
+      "generator": "Ninja",
+      "low_mem": "yes"
+    }
+  ]
+}