Add Task for publishing tekton pipeline images + yaml

.gitignore

@@ -36,3 +36,6 @@
 
 # JetBrains IDE config
 .idea
+
+# Python
+*.pyc

.ko.yaml

@@ -1,5 +1,5 @@
 baseImageOverrides:
-  # TODO(jasonhall): Use build-base in the build-pipeline path, when it's build/released.
+  # TODO(christiewilson): Use our built base image
   github.com/tektoncd/pipeline/cmd/creds-init: gcr.io/knative-nightly/github.com/knative/build/build-base:latest
   github.com/tektoncd/pipeline/cmd/git-init: gcr.io/knative-nightly/github.com/knative/build/build-base:latest
   github.com/tektoncd/pipeline/cmd/bash: busybox # image should have shell in $PATH

.ko.yaml.release

@@ -1,6 +1,7 @@
 baseImageOverrides:
-  # TODO(jasonhall): Use build-base in the build-pipeline path, when it's build/released.
+  # TODO(christiewilson): Use our built base image
   github.com/tektoncd/pipeline/cmd/creds-init: gcr.io/knative-release/github.com/knative/build/build-base:latest
   github.com/tektoncd/pipeline/cmd/git-init: gcr.io/knative-release/github.com/knative/build/build-base:latest
-  github.com/tektoncd/pipeline/cmd/bash: busybox
-  github.com/tektoncd/pipeline/cmd/gsutil: google/cloud-sdk:alpine
+  github.com/tektoncd/pipeline/cmd/bash: busybox # image should have shell in $PATH
+  github.com/tektoncd/pipeline/cmd/entrypoint: busybox # image should have shell in $PATH
+  github.com/tektoncd/pipeline/cmd/gsutil: google/cloud-sdk:alpine # image should have gsutil in $PATH

docs/resources.md

@@ -309,6 +309,8 @@ spec:
       value: gcs
     - name: location
       value: gs://some-bucket
+    - name: dir
+      value: "y" # This can have any value to be considered "true"
 ```
 
 Params that can be added are the following:
@@ -361,7 +363,7 @@ service account.
        - name: location
          value: gs://some-private-bucket
        - name: dir
-         value: "directory"
+         value: "y"
      secrets:
        - fieldName: GOOGLE_APPLICATION_CREDENTIALS
          secretName: bucket-sa

hack/release.md

@@ -1,5 +1,7 @@
 # Creating a new Tekton Pipeline release
 
+**Note: we are transitioning to a Pipelines based test and release process, see [tekton/README.md](../tekton/README.md).**
+
 The `release.sh` script automates the creation of Tekton Pipeline releases,
 either nightly or versioned ones.
 

tekton/README.md

@@ -0,0 +1,102 @@
+# Tekton Repo CI/CD
+
+We dogfood our project by using Tekton Pipelines to build, test and release Tekton Pipelines!
+
+This directory contains the [`Tasks`](https://github.com/knative/build-pipeline/blob/master/docs/tasks.md)
+and [`Pipelines`](https://github.com/knative/build-pipeline/blob/master/docs/pipelines.md) that we (will)
+use.
+
+TODO(#538): In #538 or #537 we will update [Prow](https://github.com/knative/build-pipeline/blob/master/CONTRIBUTING.md#pull-request-process)
+to invoke these `Pipelines` automatically, but for now we will have to invoke them manually.
+
+## Release Pipeline
+
+The `Tasks` which make up our release `Pipeline` are:
+
+* [`ci-images.yaml`](ci-images.yaml) - This `Task` uses [`kaniko`](https://github.com/GoogleContainerTools/kaniko)
+  to build and publish [images for the CI itself](#supporting-images), which can then be used as `steps` in
+  downstream `Tasks`
+* [`publish.yaml`](publish.yaml) - This `Task` uses [`kaniko`](https://github.com/GoogleContainerTools/kaniko)
+  to build and publish base images, and uses [`ko`](https://github.com/google/go-containerregistry/tree/master/cmd/ko)
+  to build all of the container images we release and generate the `release.yaml`
+
+### Running
+
+To run these `Pipelines` and `Tasks`, you must have Tekton Pipelines installed, either via
+[an official release](https://github.com/knative/build-pipeline/blob/master/docs/install.md)
+or [from `HEAD`](https://github.com/knative/build-pipeline/blob/master/DEVELOPMENT.md#install-pipeline).
+
+TODO(#531): Add the Pipeline, for now all we have are `Tasks` which we can invoke individually
+by creating [`TaskRuns`](https://github.com/knative/build-pipeline/blob/master/docs/taskruns.md)
+and [`PipelineResources`](https://github.com/knative/build-pipeline/blob/master/docs/resources.md).
+
+TODO(#569): Normally we'd use the image `PipelineResources` to control which image registry the images are pushed to.
+However since we have so many images, all going to the same registry, we are cheating and using a parameter
+for the image registry instead.
+
+* [`ciimages-run.yaml`](ci-images-run.yaml) - This example `TaskRun` and `PipelineResources` demonstrate
+  how to invoke `ci-images.yaml`:
+
+  ```bash
+  kubectl apply -f tekton/ci-images.yaml
+  kubectl apply -f tekton/ci-images-run.yaml
+  ```
+
+* [`publish-run.yaml`](publish-run.yaml) - This example `TaskRun` and `PipelineResources` demonstrate
+  how to invoke `publish.yaml`:
+
+  ```bash
+  kubectl apply -f tekton/publish.yaml
+  kubectl apply -f tekton/publish-run.yaml
+  ```
+
+### Authentication
+
+Users executing the publish task must be able to:
+
+* Push to the image registry (production registry is `gcr.io/tekton-releases`)
+* Write to the GCS bucket (production bucket is `gs://tekton-releases`)
+
+To be able to publish images via `kaniko` or `ko`, you must be able to push to your image registry.
+At the moment, the publish `Task` will try to use your default service account in the namespace where
+you create the `TaskRun`. If that default service account is able to push to your image registry,
+you are good to go. Otherwise, you need to use [a secret annotated with your docker registry
+credentials](https://github.com/tektoncd/pipeline/blob/master/docs/auth.md#basic-authentication-docker).
+
+TODO(#631) Ensure that we are supporting folks using credentials other than the cluster defaults; not
+sure how this will play out with publishing to our prod registry!
+
+#### Production credentials
+
+TODO(dlorenc, bobcatfish): We need to setup a group which users can be added to, as well as guidelines
+around who should be added to this group.
+
+For now, users who need access to our production registry (`gcr.io/tekton-releases`) and production
+GCS bucket (`gs://tekton-releases`) should ping @bobcatfish or @dlorenc to get added to the authorized
+users.
+
+## Supporting scripts
+
+Some supporting scripts have been written using Python 2.7:
+
+* [koparse](./koparse) - Contains logic for parsing `release.yaml` files created by `ko`
+
+## Supporting images
+
+TODO(#639) Ensure we are using the images that are published by the `Pipeline` itself.
+
+These images are built and published to be used by the release Pipeline itself.
+
+### ko image
+
+In order to run `ko`, and to be able to use a cluster's default credentials, we need an image which
+contains:
+
+* `ko`
+* `golang` - Required by `ko` to build
+* `gcloud` - Required to auth with default namespace credentials
+
+The image which we use for this is built from [tekton/ko/Dockerfile](./ko/Dockerfile).
+
+_[go-containerregistry#383](https://github.com/google/go-containerregistry/issues/383) is about publishing
+a `ko` image, which hopefully we'll be able to move it._

tekton/ci-images-run.yaml

@@ -0,0 +1,44 @@
+apiVersion: tekton.dev/v1alpha1
+kind: PipelineResource
+metadata:
+  name: tekton-pipelines
+spec:
+  type: git
+  params:
+  - name: url
+    value: https://github.com/tektoncd/pipeline # REPLACE with your own fork
+  - name: revision
+    value: master # REPLACE with your own commit
+---
+apiVersion: tekton.dev/v1alpha1
+kind: PipelineResource
+metadata:
+  name: ko-image
+spec:
+  type: image
+  params:
+   - name: url
+     value: ko-ci # Registry is provided via parameter, this is a hack see #569
+---
+apiVersion: tekton.dev/v1alpha1
+kind: TaskRun
+metadata:
+  name: publish-ci-images-run
+spec:
+  taskRef:
+    name: publish-ci-images
+  trigger:
+    type: manual
+  inputs:
+    resources:
+    - name: source
+      resourceRef:
+        name: tekton-pipelines
+    params:
+    - name: imageRegistry
+      value: gcr.io/tekton-releases # REPLACE with your own registry
+  outputs:
+    resources:
+    - name: builtKoImage
+      resourceRef:
+        name: ko-image

tekton/ci-images.yaml

@@ -0,0 +1,27 @@
+
+apiVersion: tekton.dev/v1alpha1
+kind: Task
+metadata:
+  name: publish-ci-images
+spec:
+  inputs:
+    resources:
+    - name: source
+      type: git
+    params:
+    - name: imageRegistry
+      description: TODO(#569) This is a hack to make it easy for folks to switch the registry being used by the many many image outputs
+  outputs:
+    resources:
+    - name: builtKoImage
+      type: image
+  steps:
+
+  - name: build-push-ko-image
+    image: gcr.io/kaniko-project/executor
+    command:
+    - /kaniko/executor
+    args:
+    - --dockerfile=/workspace/source/tekton/ko/Dockerfile
+    - --destination=${inputs.params.imageRegistry}/${outputs.resources.builtKoImage.url}
+    - --context=/workspace/source

tekton/ko/Dockerfile

@@ -0,0 +1,10 @@
+FROM google/cloud-sdk:latest
+
+# Install golang
+RUN curl https://dl.google.com/go/go1.12.1.linux-amd64.tar.gz > go1.12.1.tar.gz
+RUN tar -C /usr/local -xzf go1.12.1.tar.gz
+ENV PATH="${PATH}:/usr/local/go/bin"
+
+# Install ko
+ENV GOBIN=/usr/local/go/bin
+RUN go get github.com/google/go-containerregistry/cmd/ko

tekton/koparse/koparse.py

@@ -0,0 +1,116 @@
+#!/usr/bin/env python3
+
+"""
+koparse.py parses release.yaml files from `ko`
+
+The `ko` tool (https://github.com/google/go-containerregistry/tree/master/cmd/ko)
+builds images and embeds the full names of the built images in the resulting
+yaml files.
+
+This script does two things:
+
+* Parses those image names out of the release.yaml, including their digests, and
+  outputs those to stdout
+* Verifies the list of built images against an expected list, to be sure that all
+  expected images were built (and no extra images were built)
+"""
+
+import argparse
+import os
+import re
+import string
+import sys
+from typing import List
+
+
+DIGEST_MARKER = "@sha256"
+
+
+class ImagesMismatchError(Exception):
+    def __init__(self, missing: List[str], extra: List[str]):
+        self.missing = missing
+        self.extra = extra
+
+    def __str__(self):
+        errs = []
+        if self.missing:
+            errs.append("Images %s were expected but missing." % self.missing)
+        if self.extra:
+            errs.append("Images %s were present but not expected." %
+                        self.extra)
+        return " ".join(errs)
+
+
+class BadActualImageFormatError(Exception):
+    def __init__(self, image: str):
+        self.image = image
+
+    def __str__(self):
+        return "Format of image %s was unexpected, did not contain %s" % (self.image, DIGEST_MARKER)
+
+
+def parse_release(base: str, path: str) -> List[str]:
+    """Extracts built images from the release.yaml at path
+
+    Args:
+        base: The built images will be expected to start with this string,
+            other images will be ignored
+        path: The path to the file (release.yaml) that will contain the built images
+    Returns:
+        list of the images parsed from the file
+    """
+    images = []
+    with open(path) as f:
+        for line in f:
+            match = re.search(base + ".*" + DIGEST_MARKER + ":[0-9a-f]*", line)
+            if match:
+                images.append(match.group(0))
+    return images
+
+
+def compare_expected_images(expected: List[str], actual: List[str]) -> None:
+    """Ensures that the list of actual images includes only the expected images
+
+    Args:
+        expected: A list of all of the names of images that are expected to have
+            been built, including the path to the image without the digest
+        actual: A list of the names of the built images, including the path to the
+            image and the digest
+    """
+    for image in actual:
+        if DIGEST_MARKER not in image:
+            raise BadActualImageFormatError(image)
+
+    actual_no_digest = [image.split(DIGEST_MARKER)[0] for image in actual]
+
+    missing = set(expected) - set(actual_no_digest)
+    extra = set(actual_no_digest) - set(expected)
+
+    if missing or extra:
+        raise ImagesMismatchError(list(missing), list(extra))
+
+
+if __name__ == "__main__":
+    arg_parser = argparse.ArgumentParser(
+        description="Parse expected built images from a release.yaml created by `ko`")
+    arg_parser.add_argument("--path", type=str, required=True,
+                            help="Path to the release.yaml")
+    arg_parser.add_argument("--base", type=str, required=True,
+                            help="String prefix which is used to find images within the release.yaml")
+    arg_parser.add_argument("--images", type=str, required=True, nargs="+",
+                            help="List of all images expected to be built, without digests")
+    args = arg_parser.parse_args()
+
+    try:
+        images = parse_release(args.base, args.path)
+        compare_expected_images(args.images, images)
+    except (IOError, BadActualImageFormatError) as e:
+        sys.stderr.write("Error determining built images: %s\n" % e)
+        sys.exit(1)
+    except (ImagesMismatchError) as e:
+        sys.stderr.write("Expected images did not match: %s\n" % e)
+        with open(args.path) as f:
+            sys.stderr.write(f.read())
+        sys.exit(1)
+
+    print("\n".join(images))