WIP Add a CatalogTask Custom Task Type

catalogtask/.ko.yaml

@@ -0,0 +1 @@
+defaultBaseImage: alpine/git:v2.30.1

catalogtask/README.md

@@ -0,0 +1,122 @@
+# CatalogTask: Run Tasks from the Catalog without kubectl apply1
+
+## Summary
+
+This Custom Task controller demonstrates a way we could
+resolve remote tasks from the catalog.
+
+The goal of this project is simple: Users should never
+have to type `kubectl apply -f git-clone.yaml` again.
+
+## Usage
+
+Run this controller locally with `./start.sh`.
+
+You might need to edit it to make it work!
+
+Warning: This will write the file cache to your local
+disk (/tmp/ by default) for development purposes! It's
+only a single git repo though.
+
+Then apply a Run that uses a catalog task:
+
+```yaml
+# catalogtask/samples/run.yaml
+apiVersion: tekton.dev/v1alpha1
+kind: Run
+metadata:
+  name: catalog-ref-test
+  namespace: default
+spec:
+  ref:
+    apiVersion: catalogtask.tekton.dev/v1alpha1
+    kind: Task
+    name: git-clone--0.3
+  params:
+  - name: url
+    value: https://github.com/tektoncd/pipeline.git
+  workspaces:
+  - name: output
+    emptyDir: {}
+```
+
+The `git-clone` task will be fetched from the catalog
+and executed as a TaskRun with the `task.spec` in the
+`taskSpec` field and the `parameters` + `workspaces`
+passed down from the Run.
+
+`taskRun.Status.TaskRunResults` will be copied into the
+Run's `run.Status.Results` when the TaskRun completes.
+
+### TODO: This doesn't currently work in a cluster
+
+At the moment my deployment won't run in a cluster and
+I haven't figured out why. The controller binary exits
+immediately with the `-h` usage instructions printed.
+
+## Examples
+
+- [Using CatalogTasks in a Run](./samples/run.yaml)
+- [Using CatalogTasks in a Pipeline](./samples/run.yaml)
+
+## Config and Syntax
+
+### Private Catalogs
+
+By default this controller will boot up using the
+open source tekton catalog at `https://github.com/tektoncd/catalog.git`
+
+To configure a private catalog instead
+set the `CATALOG_GIT_URL` environment variable in
+the [deployment](./config/500-controller.yaml).
+
+If your private catalog requires credentials like
+SSH keys then add these as `volumeMounts` to the controller's
+template in
+[./config/500-controller.yaml](./config/500-controller.yaml).
+
+### Specifying Task Version
+
+By default using a CatalogTask will pick its latest version. E.g. this
+example will use the latest version of `github-close-issue` that it can
+find (0.2 at time of writing):
+
+```yaml
+  ref:
+    apiVersion: catalogtask.tekton.dev/v1alpha1
+    kind: Task
+    name: github-close-issue
+```
+
+The catalog stores tasks in versioned directories like this:
+
+```
+/task/github-close-issue/0.2/gihub-close-issue.yaml
+```
+
+You can specify a specific version of a Catalog Task to use
+with this slightly awkward syntax:
+
+```
+  ref:
+    apiVersion: catalogtask.tekton.dev/v1alpha1
+    kind: Task
+    name: github-close-issue--0.1 # <- Notice the --0.1 , that's the "version" syntax :/
+```
+
+### Performance
+
+For **Xtreme Performance** use an in-memory `emptyDir`
+volume as the catalog's cache. Pass the path to the cache
+to the controller with the `CACHE_PATH` env var.
+
+We do this in the [default deployment](./config/500-controller.yaml).
+
+## Next Steps
+
+- Get this running in clusters (a.k.a. why is my controller
+deployment failing ?!?!)
+- Allow an operator to pick specific versions of tasks that
+are allowed (e.g. "I only want to allow git-clone v0.2")
+- Give operators a way to fetch newer versions when the catalog
+repo is updated.

catalogtask/cmd/controller/main.go

@@ -0,0 +1,26 @@
+/*
+Copyright 2021 The Tekton Authors
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package main
+
+import (
+	"github.com/tektoncd/experimental/catalogtask/pkg/reconciler/catalogtask"
+	"knative.dev/pkg/injection/sharedmain"
+)
+
+func main() {
+	sharedmain.Main(catalogtask.ControllerName, catalogtask.NewController)
+}

catalogtask/config/200-serviceaccount.yaml

@@ -0,0 +1,5 @@
+kind: ServiceAccount
+apiVersion: v1
+metadata:
+  name: tekton-resolver
+  namespace: tekton-pipelines

catalogtask/config/201-clusterrole.yaml

@@ -0,0 +1,27 @@
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: tekton-resolver-cluster-access
+rules:
+  # Resolvers need to be able to read and write Runs,TaskRuns and Pipelines.
+  # Any of these resources may be read-from or created by a Resolver.
+  - apiGroups: ["tekton.dev"]
+    resources: ["runs", "taskruns", "pipelineruns"]
+    verbs: ["get", "list", "create", "update", "delete", "patch", "watch"]
+
+  - apiGroups: ["tekton.dev"]
+    resources: ["runs/finalizers"]
+    verbs: ["get", "list", "create", "update", "delete", "patch", "watch"]
+  - apiGroups: ["tekton.dev"]
+    resources: ["runs/status"]
+    verbs: ["get", "list", "create", "update", "delete", "patch", "watch"]
+
+  # Controller needs cluster access to leases for leader election.
+  - apiGroups: ["coordination.k8s.io"]
+    resources: ["leases"]
+    verbs: ["get", "list", "create", "update", "delete", "patch", "watch"]
+
+  # Controller needs permission to emit events associated with Run CRs.
+  - apiGroups: [""]
+    resources: ["events"]
+    verbs: ["get", "list", "create", "update", "delete", "patch", "watch"]

catalogtask/config/201-role.yaml

@@ -0,0 +1,15 @@
+kind: Role
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: tekton-resolver
+  namespace: tekton-pipelines
+rules:
+- apiGroups: [""]
+  resources: ["configmaps"]
+  verbs: ["get", "list"]
+
+# The controller needs access to these configmaps for logging information and runtime configuration.
+- apiGroups: [""]
+  resources: ["configmaps"]
+  verbs: ["get"]
+  resourceNames: ["config-logging", "config-observability", "config-leader-election"]

catalogtask/config/201-rolebinding.yaml

@@ -0,0 +1,13 @@
+kind: RoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: tekton-resolver
+  namespace: tekton-pipelines
+subjects:
+- kind: ServiceAccount
+  name: tekton-resolver
+  namespace: tekton-pipelines
+roleRef:
+  kind: Role
+  name: tekton-resolver
+  apiGroup: rbac.authorization.k8s.io

catalogtask/config/202-clusterrolebinding.yaml

@@ -0,0 +1,26 @@
+# Copyright 2021 The Tekton Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: tekton-resolver-cluster-access
+subjects:
+  - kind: ServiceAccount
+    name: tekton-resolver
+    namespace: tekton-pipelines
+roleRef:
+  kind: ClusterRole
+  name: tekton-resolver-cluster-access
+  apiGroup: rbac.authorization.k8s.io

catalogtask/config/400-controller-service.yaml

@@ -0,0 +1,27 @@
+# Copyright 2021 The Tekton Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: v1
+kind: Service
+metadata:
+  name: catalogtask-resolver
+  namespace: tekton-pipelines
+spec:
+  ports:
+    - name: http-metrics
+      port: 9090
+      protocol: TCP
+      targetPort: 9090
+  selector:
+    app.kubernetes.io/name: catalogtask-resolver

catalogtask/config/500-controller.yaml

@@ -0,0 +1,58 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: catalogtask-resolver
+  namespace: tekton-pipelines
+  labels:
+    app.kubernetes.io/name: catalogtask-resolver
+    app.kubernetes.io/component: tekton-resolver
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: catalogtask-resolver
+      app.kubernetes.io/component: tekton-resolver
+  template:
+    metadata:
+      annotations:
+        cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
+      labels:
+        app.kubernetes.io/name: catalogtask-resolver
+        app.kubernetes.io/component: tekton-resolver
+        app: catalogtask-resolver
+    spec:
+      volumes:
+      - name: config-logging
+        configMap:
+          name: config-logging
+      - name: file-cache
+        emptyDir:
+          medium: Memory
+      serviceAccountName: tekton-resolver
+      containers:
+      - name: controller
+        image: ko://github.com/tektoncd/experimental/catalogtask/cmd/controller
+        volumeMounts:
+        - name: config-logging
+          mountPath: /etc/config-logging
+        - name: file-cache
+          mountPath: /tmp/remote-task-cache
+        resources:
+          limits:
+            # in-memory file-cache volume eats in to this so the idea here is
+            # to limit the cache size to about however big the catalog repo
+            # currently is.
+            memory: 32Mi
+        env:
+        - name: SYSTEM_NAMESPACE
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.namespace
+        - name: CONFIG_LOGGING_NAME
+          value: config-logging
+        - name: METRICS_DOMAIN
+          value: tekton.dev/pipeline
+        - name: CACHE_PATH
+          value: /tmp/remote-task-cache
+        - name: CATALOG_GIT_URL
+          value: "https://github.com/tektoncd/catalog.git"

catalogtask/go.mod

@@ -0,0 +1,11 @@
+module github.com/tektoncd/experimental/catalogtask
+
+go 1.15
+
+require (
+	github.com/tektoncd/pipeline v0.21.0
+	k8s.io/api v0.19.7
+	k8s.io/apimachinery v0.19.7
+	k8s.io/client-go v0.19.7
+	knative.dev/pkg v0.0.0-20210127163530-0d31134d5f4e
+)