Skip to content

Neosperience/shipper

Repository files navigation

Shipper

CI codecov GitHub release

Ship your apps the GitOps way!

Shipper is a CLI tool that automates deploying new versions of containerized applications to GitOps repository managed by tools such as ArgoCD. Deploying such applications usually means updating a configuration file in a Git repository and writing the newly built container image tag. This file is then read by the CD tool, processed using a templater and then synchronized with a remote infrastructure environment where the application is actually deployed.

A practical example in Neosperience: applications deployed on instances of the Karavel Container Platform, which uses ArgoCD to monitor deployment configurations. To deploy a new image tag, a commit must be performed against either a Helm values.yaml file, or a Kustomize kustomization.yaml file setting a YAML key to the new value.

Shipper automates this by leveraging the native Git provider API (e.g. GitLab, GitHub, BitBucket, etc...) to atomically update the configuration file.

Supported tools

Git Providers

Templaters

  • Flat JSON (cdk.json)
  • Helm (values.yaml)
  • Kustomize (kustomization.yaml)

Usage

NAME:
   shipper - A new cli application

USAGE:
   shipper [global options] command [command options] [arguments...]

COMMANDS:
   help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --templater value, -p value                  Template system (available: "helm", "kustomize", "json") [$SHIPPER_PROVIDER]
   --repo-kind value, -t value                  Repository type (available: "gitlab", "github", "gitea", "bitbucket-cloud", "azure") (default: "gitlab") [$SHIPPER_REPO_KIND]
   --repo-branch value, -b value                Repository branch [$SHIPPER_REPO_BRANCH]
   --commit-author value, -a value              Commit author in "name <email>" format (default: "Shipper agent <[email protected]>") [$SHIPPER_COMMIT_AUTHOR]
   --commit-message value, -m value             Commit message (default: "Deploy") [$SHIPPER_COMMIT_MESSAGE]
   --container-image value, --ci value          Container image [$SHIPPER_CONTAINER_IMAGE, $SHIPPER_CONTAINER_IMAGES]
   --container-tag value, --ct value            Container tag [$SHIPPER_CONTAINER_TAG, $SHIPPER_CONTAINER_TAGS]
   --no-verify-tls                              If provided, skip X.509 certificate validation on HTTPS requests (default: false) [$SHIPPER_NO_VERIFY_TLS]
   --helm-values-file value, --hpath value      [helm] Path to values.yaml file [$SHIPPER_HELM_VALUES_FILE, $SHIPPER_HELM_VALUES_FILES]
   --helm-image-path value, --himg value        [helm] Container image path (default: "image.repository") [$SHIPPER_HELM_IMAGE_PATH, $SHIPPER_HELM_IMAGE_PATHS]
   --helm-tag-path value, --htag value          [helm] Container tag path (default: "image.tag") [$SHIPPER_HELM_TAG_PATH, $SHIPPER_HELM_TAG_PATHS]
   --kustomize-file value, --kfile value        [kustomize] Path to kustomization.yaml file [$SHIPPER_KUSTOMIZE_FILE, $SHIPPER_KUSTOMIZE_FILES]
   --json-file value, --jfile value             [json] Path to JSON file [$SHIPPER_JSON_FILE, $SHIPPER_JSON_FILES]
   --gitlab-endpoint value, --gl-uri value      [gitlab] Gitlab API endpoint, including "/api/v4" (default: "https://gitlab.com/api/v4") [$SHIPPER_GITLAB_ENDPOINT]
   --gitlab-key value, --gl-key value           [gitlab] A valid API key with commit access [$SHIPPER_GITLAB_KEY]
   --gitlab-project value, --gl-pid value       [gitlab] Project ID in "org/project" format [$SHIPPER_GITLAB_PROJECT]
   --github-endpoint value, --gh-uri value      [github] GitHub API endpoint (include "/api/v3" if using Enterprise Server) (default: "https://api.github.com") [$SHIPPER_GITHUB_ENDPOINT]
   --github-key value, --gh-key value           [github] Username/password pair in "username:password" format (use a personal access token!) [$SHIPPER_GITHUB_KEY]
   --github-project value, --gh-pid value       [github] Project ID in "org/project" format [$SHIPPER_GITHUB_PROJECT]
   --gitea-endpoint value, --ge-uri value       [gitea] Gitea API endpoint (include "/api/v1") [$SHIPPER_GITEA_ENDPOINT]
   --gitea-key value, --ge-key value            [gitea] Username/application token pair in "username:token" format [$SHIPPER_GITEA_KEY]
   --gitea-project value, --ge-pid value        [gitea] Project ID in "org/project" format [$SHIPPER_GITEA_PROJECT]
   --bitbucket-key value, --bb-key value        [bitbucket-cloud] Username/password pair in "username:password" format (use app passwords!) [$SHIPPER_GITLAB_KEY]
   --bitbucket-project value, --bb-pid value    [bitbucket-cloud] Project path in "org/project" format [$SHIPPER_GITLAB_PROJECT]
   --azure-project-id value, --az-pid value     [azure-devops] Organization and Project ID, in "org/project" format [$SHIPPER_AZURE_PROJECT_ID]
   --azure-repository-id value, --az-rid value  [azure-devops] Repository ID (if unsure, use the project ID) [$SHIPPER_AZURE_REPOSITORY_ID]
   --azure-key value, --az-key value            [azure-devops] Username/application token pair in "username:token" format [$SHIPPER_AZURE_KEY]
   --help, -h                                   show help (default: false)

The main use-case for Shipper is to be used as a CI pipeline step. In container-based CI systems like GitLab CI, GitHub Actions and alike, you can run the official container image in a step and invoke shipper with the appropriate flags.

Examples

Examples for various CI systems can be found in the examples folder. Please feel free to submit new examples if you have them!

Updating multiple images at once

Command line arguments --container-image, --container-tag, --helm-values-file, --helm-image-path, --helm-tag-path and --kustomize-file can be specified multiple times to update multiple images at once. If using the environment variables, these values are comma-separated (eg. CONTAINER_IMAGES=image1,image2).

This is an example of updating multiple images at once:

shipper -p helm --helm-values-file helm/values.yml \
  --helm-image-path image.repository --helm-tag-path image.tag --container-image img1 --container-tag tag1 \
  --helm-image-path image2.repository --helm-tag-path image2.tag --container-image img2 --container-tag tag2 \
  ... \
  --repo-branch master --commit-author "Test" ...

You can mix & match which to specify once and which to specify many times, but every tag must be specified either 1 or N times, with some exceptions where closely-related tags must always be both specified the same amount of times, for example:

  • ✔️ one --helm-values-file but many --helm-image-path (all changes will be applied to the same YAML file)
  • ✔️ many --helm-values-file and --helm-image-path (every change will be applied to a specific file, the same file can be specified multiple times)
  • ❌ 3 instances of --helm-image-path but 2 instances of --helm-values-file
  • ❌ non-equal amount of --container-image, --container-tag, --helm-image-path, --helm-tag-path

Available templaters

Helm

The Helm templater is meant to be used on Helm values.yaml files. For each image to update, you must specify a value for each of the following:

  • --helm-values-file / SHIPPER_HELM_VALUES_FILES: Path to the Helm values.yaml file to modify
  • --helm-image-path / SHIPPER_HELM_IMAGE_PATHS: YAML field in the Helm values.yaml file to modify with the specified image repository, using dot notation for nested fields (eg. image.repository)
  • --helm-tag-path / SHIPPER_HELM_TAG_PATHS: YAML field in the Helm values.yaml file to modify with the specified image tag, using dot notation for nested fields (eg. image.tag)

Kustomize

The Kustomize templater is meant to be used on kustomization.yaml files using the images list format like in this example.

To use it, you must specify the following, either once or for each image/tag pair:

  • --kustomize-file / SHIPPER_KUSTOMIZE_FILES: Path to the kustomization.yaml file to modify

JSON

The JSON templater is meant for flat JSON files such as CDK context files (cdk.json).

When using JSON, the --container-image argument will be used as the key to update in the JSON file, while the --container-tag argument will be used as the value. You will also need to specify the following value, either once or for each image/tag pair:

  • --json-file / SHIPPER_JSON_FILES: Path to the JSON file to modify

Provider notes

Azure DevOps

  • Only Git repositories are supported, not Team Foundation (TFVC).
  • When creating a personal access token for shipper, only the "Code (Read & write)" permission is needed.
  • The author string MUST be in the John Doe <[email protected]> format or the commit will ignore it and use the default (i.e. the credentials' owner).
  • You will need both a Project ID (in org/project format) and a Repository ID, if you don't know what your Repository ID is, it's probably the Project ID (without the organization). E.g. If your Project ID is my-org/my-project and you have only one repository, your Repository ID is my-project.

Bitbucket cloud

  • When creating an app password for shipper, only the permissions repository:read and repository:write are needed.
  • The author string MUST be in the John Doe <[email protected]> format or the commit will fail.
  • Bitbucket cloud integration only works with Bitbucket cloud, Bitbucket server has completely different APIs.

Gitea

  • Due to how the Commit API is implemented, calling shipper with multiple files will result in a multiple commits, one per modified file.

GitHub

  • When creating a personal access token for shipper, only the permissions repo is needed.
  • The author string MUST be in the John Doe <[email protected]> format or the commit will fail.
  • The GitHub Cloud API endpoint is https://api.github.com, however GitHub Enterprise Server will have something more akin to https://HOSTNAME/api/v3
  • Due to how the Commit API is implemented, calling shipper with multiple files will result in a multiple commits, one per modified file.

GitLab

  • When creating a project access token for shipper, only the permission api is needed. (Role depends on your branch permissions, eg. protected branches)

Contributing

Found a bug or need a new feature? Open an issue and discuss it with the team!. Don't forget to check out our contribution guide for more information!

Releasing

See our release guide

License

Copyright 2022 Neosperience S.P.A.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Logo by joezhuang on Vecteezy