Skip to content

Commit

Permalink
Backport release "scripts" changes from main…
Browse files Browse the repository at this point in the history
… as we switch to ghcr.io, we need those to be updated to work as they
are referered during releases.

Signed-off-by: Vincent Demeester <[email protected]>
  • Loading branch information
vdemeester committed Oct 15, 2024
1 parent ea65f15 commit 5931ab7
Show file tree
Hide file tree
Showing 5 changed files with 184 additions and 35 deletions.
3 changes: 1 addition & 2 deletions tekton/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ consumers of a project. In that case we'll make a patch release. To make one:

## Nightly releases

[The nightly release pipeline](release-pipeline-nightly.yaml) is
[The nightly release pipeline](release-pipeline.yaml) is
[triggered nightly by Tekton](https://github.com/tektoncd/plumbing/tree/main/tekton).

This Pipeline uses:
Expand Down Expand Up @@ -110,7 +110,6 @@ Apply the tasks from the `pipeline` repo:
# Apply the Tasks and Pipelines we use from this repo
kubectl apply -f tekton/publish.yaml
kubectl apply -f tekton/release-pipeline.yaml
kubectl apply -f tekton/release-pipeline-nightly.yaml

# Apply the resources - note that when manually releasing you'll re-apply these
kubectl apply -f tekton/resources.yaml
Expand Down
67 changes: 67 additions & 0 deletions tekton/bugfix-release.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
#!/usr/bin/env bash
set -eu -o pipefail

RELEASE_BRANCH=${1:-$(git rev-parse --abbrev-ref HEAD)}
shift

echo "> Make sure our remotes are up-to-date"
git fetch -p --all

TEKTON_RELEASE_GIT_SHA=$(git rev-parse "${RELEASE_BRANCH}")
TEKTON_OLD_VERSION=$(git describe --tags --abbrev=0 "${TEKTON_RELEASE_GIT_SHA}")
TEKTON_OLD_VERSION_COMMIT_SHA=$(git rev-list -n 1 "${TEKTON_OLD_VERSION}")
TEKTON_RELEASE_NAME=$(gh release view "${TEKTON_OLD_VERSION}" --json name | jq .name | sed -e 's/.*\\"\(.*\)\\"\"/\1/')

if [[ "${TEKTON_RELEASE_GIT_SHA}" == "${TEKTON_OLD_VERSION_COMMIT_SHA}" ]]; then
echo "> No new commit in ${RELEASE_BRANCH} (${TEKTON_RELEASE_GIT_SHA}==${TEKTON_OLD_VERSION_COMMIT_SHA})"
exit 0
fi

TEKTON_VERSION=$(echo ${TEKTON_OLD_VERSION} | awk -F. -v OFS=. '{$NF += 1 ; print}')

echo "> Old version : ${TEKTON_OLD_VERSION}"
echo "> Old version commit : ${TEKTON_OLD_VERSION_COMMIT_SHA}"
echo "> New version : ${TEKTON_VERSION}"
echo "> New version commit: ${TEKTON_RELEASE_GIT_SHA}"
echo "> Tekton Release Name: ${TEKTON_RELEASE_NAME}"

# Might be overkill
git --no-pager diff "${TEKTON_OLD_VERSION_COMMIT_SHA}" "${TEKTON_RELEASE_GIT_SHA}"

cat <<EOF > workspace-template.yaml
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
EOF

echo "> Starting the release pipeline"
tkn pipeline start pipeline-release \
--serviceaccount=release-right-meow \
--param=gitRevision="${TEKTON_RELEASE_GIT_SHA}" \
--param=serviceAccountPath=release.json \
--param=versionTag="${TEKTON_VERSION}" \
--param=releaseBucket=gs://tekton-releases/pipeline \
--param=releaseAsLatest="false" \
--workspace name=release-secret,secret=release-secret \
--workspace name=workarea,volumeClaimTemplateFile=workspace-template.yaml --use-param-defaults --pipeline-timeout 3h --showlog

RELEASE_FILE=https://storage.googleapis.com/tekton-releases/pipeline/previous/${TEKTON_VERSION}/release.yaml
CONTROLLER_IMAGE_SHA=$(curl $RELEASE_FILE | egrep 'gcr.io.*controller' | cut -d'@' -f2)
REKOR_UUID=$(rekor-cli search --sha $CONTROLLER_IMAGE_SHA | grep -v Found | head -1)
echo -e "CONTROLLER_IMAGE_SHA: ${CONTROLLER_IMAGE_SHA}\nREKOR_UUID: ${REKOR_UUID}"

echo "> Starting the release-draft pipeline"
tkn pipeline start release-draft \
--workspace name=shared,volumeClaimTemplateFile=workspace-template.yaml \
--workspace name=credentials,secret=release-secret \
-p package="tektoncd/pipeline" \
-p git-revision="${TEKTON_RELEASE_GIT_SHA}" \
-p release-tag="${TEKTON_VERSION}" \
-p previous-release-tag="${TEKTON_OLD_VERSION}" \
-p release-name="${TEKTON_RELEASE_NAME}" \
-p bucket="gs://tekton-releases/pipeline" \
-p rekor-uuid="$REKOR_UUID" \
--showlog
85 changes: 64 additions & 21 deletions tekton/publish.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@ spec:
- name: images
description: List of cmd/* paths to be published as images
default: "controller webhook entrypoint nop workingdirinit resolvers sidecarlogresults events"
- name: koExtraArgs
description: Extra args to be passed to ko
default: "--preserve-import-paths"
- name: versionTag
description: The vX.Y.Z version that the artifacts should be tagged with (including `v`)
- name: imageRegistry
Expand All @@ -22,6 +25,9 @@ spec:
- name: imageRegistryRegions
description: The target image registry regions
default: "us eu asia"
- name: imageRegistryUser
description: Username to be used to login to the container registry
default: "_json_key"
- name: releaseAsLatest
description: Whether to tag and publish this release as Pipelines' latest
default: "true"
Expand All @@ -46,41 +52,47 @@ spec:
env:
- name: "PROJECT_ROOT"
value: "$(workspaces.source.path)"
- name: CONTAINER_REGISTY_CREDENTIALS
- name: CONTAINER_REGISTRY_CREDENTIALS
value: "$(workspaces.release-secret.path)/$(params.serviceAccountPath)"
- name: CONTAINER_REGISTRY
value: "$(params.imageRegistry)/$(params.imageRegistryPath)"
- name: IMAGE_REGISTRY_PATH
value: "$(params.imageRegistryPath)"
- name: CONTAINER_REGISTRY_USER
value: "$(params.imageRegistryUser)"
- name: REGIONS
value: "$(params.imageRegistryRegions)"
- name: OUTPUT_RELEASE_DIR
value: "$(workspaces.output.path)/$(params.versionTag)"
- name: KO_EXTRA_ARGS
value: "$(params.koExtraArgs)"
results:
# IMAGES result is picked up by Tekton Chains to sign the release.
# See https://github.com/tektoncd/plumbing/blob/main/docs/signing.md for more info.
- name: IMAGES
steps:

- name: container-registy-auth
image: gcr.io/go-containerregistry/crane:debug
- name: container-registry-auth
image: cgr.dev/chainguard/crane:latest-dev@sha256:b33c82b22a6cfb21e3db968fba5f426461f7540d7fa37048e2a6ffb9aaca7f19
script: |
#!/busybox/sh
#!/bin/sh
set -ex
# Login to the container registry
DOCKER_CONFIG=$(cat ${CONTAINER_REGISTY_CREDENTIALS} | \
crane auth login -u _json_key --password-stdin $(params.imageRegistry) 2>&1 | \
DOCKER_CONFIG=$(cat ${CONTAINER_REGISTRY_CREDENTIALS} | \
crane auth login -u ${CONTAINER_REGISTRY_USER} --password-stdin $(params.imageRegistry) 2>&1 | \
sed 's,^.*logged in via \(.*\)$,\1,g')
# Auth with account credentials for all regions.
for region in ${REGIONS}
do
HOSTNAME=${region}.$(params.imageRegistry)
cat ${CONTAINER_REGISTY_CREDENTIALS} | crane auth login -u _json_key --password-stdin ${HOSTNAME}
cat ${CONTAINER_REGISTRY_CREDENTIALS} | crane auth login -u ${CONTAINER_REGISTRY_USER} --password-stdin ${HOSTNAME}
done
cp ${DOCKER_CONFIG} /workspace/docker-config.json
- name: create-ko-yaml
image: golang:1.18.7
image: cgr.dev/chainguard/go:latest-dev@sha256:35dc4adbb3b6fadafd60d0a004d06d706f2f6ed1511f5c24f22f92f8fe94f783
script: |
#!/bin/sh
set -ex
Expand All @@ -93,13 +105,21 @@ spec:
# Change to directory with vendor/
cd ${PROJECT_ROOT}
COMBINED_BASE_IMAGE_BASE=${CONTAINER_REGISTRY}
# If the IMAGE_REGISTRY_PATH does not already includes the package, add it
# Package looks like github.com/<org>/<repo>
# Path may look like "tekton-releases" or "tektoncd/pipeline"
if [[ ! "$(params.package)" == "github.com/${IMAGE_REGISTRY_PATH}" ]]; then
COMBINED_BASE_IMAGE_BASE=${COMBINED_BASE_IMAGE_BASE}/${IMAGE_REGISTRY_PATH}
fi
# Combine Distroless with a Windows base image, used for the entrypoint image.
# Distroless is pinned to the last version based on Alpine 3.18. Newer versions are based on Alpine 3.19_alpha20230901.
COMBINED_BASE_IMAGE=$(go run ./vendor/github.com/tektoncd/plumbing/cmd/combine/main.go \
cgr.dev/chainguard/static@sha256:67a1b00e0134e2b3a614c7198a26f7deed9d11b7acad4d52c79c0cfd47a2eae7 \
mcr.microsoft.com/windows/nanoserver:ltsc2019 \
mcr.microsoft.com/windows/nanoserver:ltsc2022 \
${CONTAINER_REGISTRY}/$(params.package)/combined-base-image:latest)
${COMBINED_BASE_IMAGE_BASE}/combined-base-image:latest)
# NOTE: Make sure this list of images to use the combined base image is in sync with what's in test/presubmit-tests.sh's 'ko_resolve' function.
cat <<EOF > /workspace/.ko.yaml
Expand All @@ -115,7 +135,7 @@ spec:
cat /workspace/.ko.yaml
- name: run-ko
image: gcr.io/tekton-releases/dogfooding/ko@sha256:37f6b1481af35d8f0d194062867df45842d55c467b3c40e6561b1a1defea1cc0
image: gcr.io/tekton-releases/dogfooding/ko@sha256:bdcd596b40583f6f8316745e27d7ff327a0756fb6aee1251dfc0bdbd01b26c88
env:
- name: KO_DOCKER_REPO
value: $(params.imageRegistry)/$(params.imageRegistryPath)
Expand Down Expand Up @@ -159,46 +179,69 @@ spec:
# The real "tagging" will happen with the "create-release" pipeline.
git tag $(params.versionTag)
ko resolve --platform=$(params.platforms) --preserve-import-paths -t $(params.versionTag) -R -f ${PROJECT_ROOT}/config/ > $OUTPUT_RELEASE_DIR/release.yaml
ko resolve \
--image-label=org.opencontainers.image.source=https://$(params.package) \
--platform=$(params.platforms) \
-t $(params.versionTag) \
-R ${KO_EXTRA_ARGS} \
-f ${PROJECT_ROOT}/config/ > $OUTPUT_RELEASE_DIR/release.yaml
# Publish images and create release.notags.yaml
# This is useful if your container runtime doesn't support the `image-reference:tag@digest` notation
# This is currently the case for `cri-o` (and most likely others)
ko resolve --platform=$(params.platforms) --preserve-import-paths -R -f ${PROJECT_ROOT}/config/ > $OUTPUT_RELEASE_DIR/release.notags.yaml
ko resolve \
--image-label=org.opencontainers.image.source=https://$(params.package) \
--platform=$(params.platforms) \
-R ${KO_EXTRA_ARGS} \
-f ${PROJECT_ROOT}/config/ > $OUTPUT_RELEASE_DIR/release.notags.yaml
# Rewrite "devel" to params.versionTag
sed -i -e 's/\(pipeline.tekton.dev\/release\): "devel"/\1: "$(params.versionTag)"/g' -e 's/\(app.kubernetes.io\/version\): "devel"/\1: "$(params.versionTag)"/g' -e 's/\(version\): "devel"/\1: "$(params.versionTag)"/g' ${OUTPUT_RELEASE_DIR}/release.yaml
sed -i -e 's/\(pipeline.tekton.dev\/release\): "devel"/\1: "$(params.versionTag)"/g' -e 's/\(app.kubernetes.io\/version\): "devel"/\1: "$(params.versionTag)"/g' -e 's/\(version\): "devel"/\1: "$(params.versionTag)"/g' ${OUTPUT_RELEASE_DIR}/release.notags.yaml
- name: koparse
image: gcr.io/tekton-releases/dogfooding/koparse@sha256:5945f709f5533347e2fac2f7e757a2acde2ce25418a7193489bf49027aa0497f
image: gcr.io/tekton-releases/dogfooding/koparse@sha256:6b70f2d6fc1cc7849c5e65dcf404ee153653055799ceea511935bba7a27d3c44
script: |
set -ex
IMAGES_PATH=${CONTAINER_REGISTRY}/$(params.package)
# Find "--preserve-import-paths" in a list of args
function find_preserve_import_path() {
for arg in $@; do
if [[ "$arg" == "--preserve-import-paths" ]]; then
return 0
fi
done
return 1
}
# If "--preserve-import-paths" is used, include "package" in the expected path
find_preserve_import_path \
$(echo $KO_EXTRA_ARGS) && \
PRESERVE_IMPORT_PATH="--preserve-path" || \
PRESERVE_IMPORT_PATH="--no-preserve-path"
for cmd in $(params.images)
do
IMAGES="${IMAGES} ${IMAGES_PATH}/cmd/${cmd}:$(params.versionTag)"
IMAGES="${IMAGES} $(params.package)/cmd/${cmd}:$(params.versionTag)"
done
# Parse the built images from the release.yaml generated by ko
koparse \
--path $OUTPUT_RELEASE_DIR/release.yaml \
--base ${IMAGES_PATH} --images ${IMAGES} > /workspace/built_images
--base $(params.package) \
--container-registry ${CONTAINER_REGISTRY} \
--images ${IMAGES} \
${PRESERVE_IMPORT_PATH} > /workspace/built_images
- name: tag-images
image: gcr.io/go-containerregistry/crane:debug
image: cgr.dev/chainguard/crane:latest-dev@sha256:b33c82b22a6cfb21e3db968fba5f426461f7540d7fa37048e2a6ffb9aaca7f19
script: |
#!/busybox/sh
#!/bin/sh
set -ex
# Setup docker-auth
DOCKER_CONFIG=~/.docker
mkdir -p ${DOCKER_CONFIG}
cp /workspace/docker-config.json ${DOCKER_CONFIG}/config.json
REGIONS="us eu asia"
# Tag the images and put them in all the regions
for IMAGE in $(cat /workspace/built_images)
do
Expand Down
30 changes: 22 additions & 8 deletions tekton/release-cheat-sheet.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,7 @@ the pipelines repo, a terminal window and a text editor.

1. `cd` to root of Pipelines git checkout.

1. Select the commit you would like to build the release from (NOTE: the commit is full (40-digit) hash.)
- Select the most recent commit on the ***main branch*** if you are cutting a major or minor release i.e. `x.0.0` or `0.x.0`
- Select the most recent commit on the ***`release-<version number>x` branch***, e.g. [`release-v0.47.x`](https://github.com/tektoncd/pipeline/tree/release-v0.47.x) if you are patching a release i.e. `v0.47.2`.
1. [Install kustomize](https://kubectl.docs.kubernetes.io/installation/kustomize) if you haven't already.

1. Ensure the correct version of the release pipeline is installed on the cluster:

Expand All @@ -24,6 +22,13 @@ the pipelines repo, a terminal window and a text editor.

```bash
TEKTON_VERSION=# Example: v0.21.0
```

- Select the commit you would like to build the release from (NOTE: the commit is full (40-digit) hash.)
- Select the most recent commit on the ***main branch*** if you are cutting a major or minor release i.e. `x.0.0` or `0.x.0`
- Select the most recent commit on the ***`release-<version number>x` branch***, e.g. [`release-v0.47.x`](https://github.com/tektoncd/pipeline/tree/release-v0.47.x) if you are patching a release i.e. `v0.47.2`.

```bash
TEKTON_RELEASE_GIT_SHA=# SHA of the release to be released
```

Expand All @@ -47,21 +52,30 @@ the pipelines repo, a terminal window and a text editor.
```
1. Execute the release pipeline (takes ~45 mins).
**The minimum required tkn version is v0.30.0 or later**
**If you are back-porting include this flag: `--param=releaseAsLatest="false"`**
```bash
tkn --context dogfooding pipeline start pipeline-release \
--serviceaccount=release-right-meow \
--param=gitRevision="${TEKTON_RELEASE_GIT_SHA}" \
--param=serviceAccountPath=release.json \
--param=versionTag="${TEKTON_VERSION}" \
--param=releaseBucket=gs://tekton-releases/pipeline \
--param package=github.com/tektoncd/pipeline \
--param gitRevision="${TEKTON_RELEASE_GIT_SHA}" \
--param imageRegistry=ghcr.io \
--param imageRegistryPath=tektoncd/pipeline \
--param imageRegistryRegions="" \
--param imageRegistryUser=tekton-robot \
--param serviceAccountPath=release.json \
--param serviceAccountImagesPath=credentials \
--param versionTag="${TEKTON_VERSION}" \
--param releaseBucket=gs://tekton-releases/pipeline \
--param koExtraArgs="" \
--workspace name=release-secret,secret=release-secret \
--workspace name=release-images-secret,secret=ghcr-creds \
--workspace name=workarea,volumeClaimTemplateFile=workspace-template.yaml \
--tasks-timeout 2h \
--pipeline-timeout 3h
```
Accept the default values of the parameters (except for "releaseAsLatest" if backporting).
Expand Down
Loading

0 comments on commit 5931ab7

Please sign in to comment.