-
Notifications
You must be signed in to change notification settings - Fork 23
Template Experiment
This page describes an experiment conducted by moritz that introduced templating to generated JSON for the GoCD JSON configuration plugin.
A similar setup was created for the GRiSP project, using Mustache templating, and Erlang for pipeline creation by nextl00p. See the generation logic, the generated pipelines and the build scripts.
There are two git repositories:
-
go-pipelines-sourcecontains templates written in the jsonnet data template language, and jsonnet files. It also contains a git repository that callsjsonneton each source file to generate full JSON for GoCD pipelines -
go-pipelines-generatedcontains only generated files. It is filled by the shell script fromgo-pipelines-source. This repository is used as a configuration repository in GoCD.
In addition, there is a GoCD pipeline called gocd-pipelines that checks out go-pipelines-source, runs the generation script, and pushes the result to go-pipelines-generated.
Thus, the workflow for creating a new pipeline is:
- Add a source file to
go-pipelines-source, git add, git commit, git push. - The pipeline
gocd-pipelinesruns, creates the full JSON<yourpipeline>.gopipeline.jsonfiles in thego-pipelines-generatedrepository, and commits and pushes the generated files - GoCD polls the config repository
go-pipelines-generatedand learns about the new pipeline.
The vast majority of our pipelines follow the same scheme:
- Build a package (Debian or Python)
- upload to a mirror for test
- (optionally) deploy to a test environment
- (optionally) run end-to-end tests
- manual approval
- upload to a mirror for production
- (optionally) deploy to production
but there is a lot of variability in the details: some packages need to be built on just one Debian distribution, others on multiple. Some are deployed, some are just available to be drawn as dependencies. Some have e2e/system tests in the test environment, some don't. Some Python packages aren't actually moved to production, because they are included in other Debian packages (fat-packaging using dh-virtualenv).
We started with roughly 140 pipelines, 120 of which used GoCD's built-in pipelines.
The 120 templated pipelines were all replaced by pipelines generated from jsonnet templates.
This reduced the number of templates from 12 to 3, and reduced the number of lines of GoCD configuration by 53%. In addition, we managed to reduce the number of stages by 96, which previously were only filled with placeholders, because GoCD templates don't allow conditional inclusion of stages.
These were the features that allowed me to reduce both the number of templates and lines of code generally:
- being able to template everything, including resources (and not just as string, as arrays)
- conditionals that allowed me to exclude tasks or even stages where not necessary, thus avoiding either dummy tasks or duplication
- the ability to use (computed) defaults meant less repetition when instantiating the templates
- with list comprehensions I was able to pass in a list of Debian distributions to build packages on, and have the template generate one job per distribution, and later in the pipeline one job per distribution that d uploads each package to the corresponding repository.
Having a fully-fledged template system between the hand-edited files and what GoCD consumes adds a lot of flexibility that GoCD's built-in template does not provide, but that is very beneficial for real-world uses cases.