From 34f2bc0848f7fc7f3a40048fdfa66ccf59ea5687 Mon Sep 17 00:00:00 2001 From: Jonathan Matthews Date: Fri, 13 Oct 2023 17:06:40 +0100 Subject: [PATCH] content: driving gitlab ci/cd pipelines with CUE This change adds a new piece of content which shows how to manage a GitLab repo's CI/CD pipeline file in CUE, instead of YAML. Because GitLab still needs to see a YAML file serialised in the repo, the guide includes a CUE _tool that turns the CUE back into YAML on demand. It also includes a schema for the pipeline's representation, but this schema is pretty lightweight and could do with improving. Part of the problem is that I had to manually write the schema, as GitLab's JSONSchema schema currently confuses `cue import`. I opened cue-lang/cue#2654 to track this. Closes #19 --- README.md | 2 + WIP_gitlab_ci/README.md | 366 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 368 insertions(+) create mode 100644 WIP_gitlab_ci/README.md diff --git a/README.md b/README.md index a03a440..04b9985 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,8 @@ integrating with 3rd-party tools, services, and systems. - 003: [Controlling Kubernetes with CUE](003_kubernetes_tutorial/README.md) - 004: [Managing Mythic Beasts DNS zones with CUE](004_mythic_beasts_dns/README.md) +- WIP: [Driving GitLab CI/CD pipelines with CUE](WIP_gitlab_ci/README.md) + ## Contributing First please read the [style guide](000_template/README.md), and then use diff --git a/WIP_gitlab_ci/README.md b/WIP_gitlab_ci/README.md new file mode 100644 index 0000000..7fa5a71 --- /dev/null +++ b/WIP_gitlab_ci/README.md @@ -0,0 +1,366 @@ +# Driving GitLab CI/CD pipelines with CUE +by [Jonathan Matthews](https://jonathanmatthews.com) + +This guide explains how to convert a GitLab CI/CD pipeline file from YAML to +CUE, check its contents are valid, and then use CUE's tooling layer to +regenerate YAML. + +This is useful because it allows you to switch to CUE as a source of truth for +GitLab pipelines and perform client-side validation, without GitLab needing to +know you're managing your pipelines with CUE. + +## Prerequisites + +- You have + [CUE installed](https://alpha.cuelang.org/docs/introduction/installation/) + locally. This allows you to run `cue` commands. +- You have a GitLab pipeline file. + - The example shown throughout this guide uses the state of a specific commit + from the + [Flockademic repository](https://gitlab.com/Flockademic/Flockademic/-/blob/8efcea927b10c2773790fe78bb858905a75cf3ef/.gitlab-ci.yml) + on gitlab.com, as linked from + [GitLab's documentation pages](https://docs.gitlab.com/ee/ci/examples/end_to_end_testing_webdriverio/index.html), + but you don't need to use that repository in any way.\ + It is used here as it represents a reasonably complex example of a GitLab + pipeline file. +- You have [`git` installed](https://git-scm.com/downloads). + + +## Steps + +### Convert YAML pipeline to CUE + +#### :arrow_right: Begin with a clean git state + +Change directory into the root of the repository that contains your GitLab +pipeline file, and ensure you start this process with a clean git state, with +no modified files. For example: + +:computer: `terminal` +```sh +cd Flockademic # our example repository +git status # should report "working tree clean" +``` + +#### :arrow_right: Initialise a CUE module + +Initialise a CUE module named after the organisation and repository you're +working with. For example: + +:computer: `terminal` +```sh +cue mod init gitlab.com/Flockademic/Flockademic +``` + +#### :arrow_right: Import YAML pipeline + +Use `cue` to import your YAML pipeline file: + +:computer: `terminal` +```sh +cue import .gitlab-ci.yml --with-context -p gitlab -f -l pipelines: -l 'strings.TrimSuffix(path.Base(filename),path.Ext(filename))' -o gitlab-ci.cue +``` + +If your project uses a different name for your pipeline file then use that name +in the above command, and throughout this guide. + +Check that a CUE file has been created from your pipeline file. For example: + +:computer: `terminal` +```sh +ls {,.}*gitlab-ci* +``` + +Your output should look similar to this, with a matching YAML and CUE file: + +```text +gitlab-ci.cue .gitlab-ci.yml +``` + +Observe that your file has been imported into the `pipelines` struct at a +location derived from its original file name, by running: + +:computer: `terminal` +```sh +head gitlab-ci.cue +``` + +The output should reflect your pipeline. In our example: + +```text +package gitlab +pipelines: ".gitlab-ci": { + image: "node:8.10" + + stages: [ + "prepare", + "test", + "build-backend", + "deploy-backend", +``` + +#### :arrow_right: Store CUE pipelines in a dedicated directory + +Create a directory called `gitlab` to hold your CUE-based GitLab pipeline +files. For example: + +:computer: `terminal` +```sh +mkdir -p internal/ci/gitlab +``` + +You may change the hierarchy and naming of `gitlab`'s **parent** directories to +suit your repository layout. If you do so, you will need to adapt some commands +and CUE code as you follow this guide. + +Move the newly-created CUE pipeline file into its dedicated directory. For example: + +:computer: `terminal` +```sh +mv gitlab-ci.cue internal/ci/gitlab +``` + +### Validate pipeline + +#### :arrow_right: Create a pipeline schema + + + +Create a basic CUE schema for GitLab pipelines, adapted from [GitLab's CI/CD +documentation](https://docs.gitlab.com/ee/ci/yaml/index.html), and place it in +the `internal/ci/gitlab` directory: + +:floppy_disk: `internal/ci/gitlab/gitlab.cicd.pipeline.schema.cue` + +```CUE +package gitlab + +_globalKeywords: ["default", "include", "stages", "variables", "workflow"] +_#job: _ +#Pipeline: { + default?: { + after_script?: _ + artifacts?: _ + before_script?: _ + cache?: _ + hooks?: _ + id_tokens?: _ + image?: _ + interruptible?: _ + retry?: _ + services?: _ + tags?: _ + timeout?: _ + } + include?: _ + stages?: [...string] + variables?: _ + workflow?: _ + [and([ for _, v in _globalKeywords {!=v}])]: _#job +} +``` + +| :grey_exclamation: Info :grey_exclamation: | +|:---------------------------------------------- | +| It would be great if we could use [GitLab's authoritative pipeline schema](https://gitlab.com/gitlab-org/gitlab/-/raw/master/app/assets/javascripts/editor/schema/ci.json), here. Unfortunately, CUE's JSONSchema support can't import it currently. This is being tracked in [CUE Issue #2654](https://github.com/cue-lang/cue/issues/2654), and this guide should be updated once the issue is resolved. + +#### :arrow_right: Apply the schema + +We need to tell CUE to apply the schema to the pipeline. + +To do this we'll create a file at `internal/ci/gitlab/pipelines.cue` in our +example. + +However, if your earlier pipeline import *already* created a file with that +same path and name, then simply select a different CUE filename that *doesn't* +already exist. Place the file in the `internal/ci/gitlab/` directory. + +:floppy_disk: `internal/ci/gitlab/pipelines.cue` + +``` +package gitlab + +// each member of the pipelines struct must be a valid #Pipeline +pipeline: [_]: #Pipeline +``` + +### Generate YAML from CUE + +#### :arrow_right: Create a CUE tool file + +Create a CUE "tool" file in `internal/ci/gitlab/` and adapt the +element commented with `TODO`: + +:floppy_disk: `internal/ci/gitlab/ci_tool.cue` +```CUE +package gitlab + +import ( + "path" + "encoding/yaml" + "tool/file" +) + +_goos: string @tag(os,var=os) + +// Regenerate pipeline files +command: regenerate: { + pipeline_files: { + // TODO: update _toolFile to reflect the directory hierarchy containing this file. + // TODO: update _pipelineDir to reflect the directory containing your pipeline file. + let _toolFile = "internal/ci/gitlab/ci_tool.cue" + let _pipelineDir = path.FromSlash(".", path.Unix) + let _donotedit = "Code generated by \(_toolFile); DO NOT EDIT." + + for _pipelineName, _pipelineConfig in pipelines + let _pipelineFile = _pipelineName + ".yml" + let _pipelinePath = path.Join([_pipelineDir, _pipelineFile]) { + let delete = { + "Delete \(_pipelinePath)": file.RemoveAll & {path: _pipelinePath} + } + delete + create: file.Create & { + $after: delete + filename: _pipelinePath + contents: "# \(_donotedit)\n\n\(yaml.Marshal(_pipelineConfig))" + } + } + } +} +``` + +Make the modifications indicated by the `TODO` comments. + +This tool will export your CUE-based pipeline back into its required YAML file, +on demand. + +#### :arrow_right: Test the CUE tool file + +With the modified `ci_tool.cue` file in place, check that the `regenerate` +command is available **from a shell sitting at the repository root**. For +example: + +:computer: `terminal` +```sh +cd $(git rev-parse --show-toplevel) # make sure we're sitting at the repository root +cue help cmd regenerate ./internal/ci/gitlab # the "./" prefix is required +``` + +The output of the `cue help` command **must** begin with the following: + +```text +Regenerate all pipeline files + +Usage: + cue cmd regenerate [flags] +[... output continues ...] +``` + +| :exclamation: WARNING :exclamation: | +|:--------------------------------------- | +| If you *don't* see the usage explanation for the `regenerate` command (or if you receive an error message) then your tool file isn't set up as CUE requires. Double check the contents of the `ci_tool.cue` file and the modifications you made to it, as well as its location in the repository. Ensure the filename is *exactly* `ci_tool.cue`. Make sure you've followed all the steps in this guide, and that you invoked the `cue help` command from the root of the repository. + +#### :arrow_right: Regenerate the YAML pipeline file + +Run the `regenerate` command to produce a YAML pipeline file from CUE. For +example: + +:computer: `terminal` +```sh +cue cmd regenerate ./internal/ci/gitlab # the "./" prefix is required +``` + +#### :arrow_right: Audit changes to the YAML pipeline file + +Check that your YAML pipeline file has a single *material* change from the +original: + +:computer: `terminal` +```sh +git diff .gitlab-ci.yml +``` + +Your output should look similar to the following example: + +```diff +diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml +index d0eaf801..d0a309e3 100644 +--- a/.gitlab-ci.yml ++++ b/.gitlab-ci.yml +@@ -1,5 +1,6 @@ +-image: node:8.10 ++# Code generated by internal/ci/gitlab/ci_tool.cue; DO NOT EDIT. + ++image: node:8.10 + stages: + - prepare + - test +[ ... output continues ... ] +``` + +The main change in each YAML file is the addition of a header that warns the +reader not to edit the file directly. + +Your diff might also contain some YAML reformatting (with the number of leading +spaces having been changed in nested structures) but this won't make a +difference to the underlying meaning of the file. + +Additionally, any comments in the original YAML file will now be found *only* +in the CUE source file - which is important as that's the only file that you'll +be manually changing, from now on. + +#### :arrow_right: Add and commit files to git + +Add your files to git. For example: + +:computer: `terminal` +```sh +git add .gitlab-ci.yml internal/ci/gitlab/ cue.mod/module.cue +``` + +Make sure to include your slightly modified YAML pipeline file, wherever you +store it, along with all the new files in `internal/ci/gitlab/` and your +`cue.mod/module.cue` file. + +Commit your files to git, with an appropriate commit message: + +:computer: `terminal` +```sh +git commit -m "ci: create CUE sources for GitLab CI/CD pipelines" +``` + +## Conclusion + +**Well done - your GitLab CI/CD pipeline file has been imported into CUE!** + +It can now be managed using CUE, leading to safer and more predictable changes. +The use of a schema to check your pipeline means that you will catch and fix +certain types of mistake earlier than before, without waiting for the slow "git +add/commit/push; check if CI fails" cycle. + +From now on, each time you make a change to a CUE pipeline file, immediately +regenerate the YAML files required by GitLab CI/CD, and commit your changes to +all the CUE and YAML files. For example: + +:computer: `terminal` +```sh +cue cmd regenerate ./internal/ci/gitlab/ # the "./" prefix is required +git add .gitlab-ci.yml internal/ci/gitlab/ +git commit -m "ci: added new release pipeline" # example message +```