Cicd delete old prs (kubeflow#575)

* Define a script to close obsolete PRs to update an application.

* In the event there are multiple PRs open to update a Kubeflow application
  we want to close the older PRs; so there is a single open PR updating
  the application to the newest code.

Related to kubeflow#571

* Define a script to close obsolete PRs to update an application.

* In the event there are multiple PRs open to update a Kubeflow application
  we want to close the older PRs; so there is a single open PR updating
  the application to the newest code.

  Related to kubeflow#571

* Setup a def namespace for use with apps-cd.

* Update update_kf_apps.py to close old PRs on each sync.

* Bake the source code into the docker image rather than using a wrapper
  script to sync the code from git.

  * Sync'ing the code from git became to difficult to reason about once
    we start splitting the source code across multiple repositories
    * We now depend on github/kubeflow/code-intelligence for utilities
      for working with GitHub Apps

    * Using a docker image also ensures we don't get broken suddenly when
      new changes are in place

    * In the future we could use github actions to automate updating the
      deployment on postsubmits

* Turn app-pipeline.template.yaml into a ConfigMap
  * This allows better versioning
  * We can rely on kustomize to create a configmap with a hash based on the
    contents
  * kustomize will then reference the config map using its hash. As
    a result a rolling update is triggered whenever the hash contents changes.
  * This makes it easier to handle rollous and updates.

Define a dev instance of the update KF apps infrastructure to facilitate development
  * Use profiles in skaffold.

  * update_kf_apps.py in dev uses a config map now to ubtain app-pipeline.template.yaml
    rather than fetching it from git

  * This makes it much easier to test changes in the dev instance

Fix a bunch of bugs preventing update_kf_apps.py from working
   * Update requirements.txt with a bunch of missing packages.
   * Fix some imports in update_kf_apps.py

  * Need to set resource requests for the build pods otherwise builds get
    CPU starved and take forever.

Miscellaneous

* Create a tool to copy secrets between namespaces from GCS

* Fix lint.

* Due to kubeflow#460 we need to disable pylint.

apps-cd/Dockerfile

@@ -1,3 +1,7 @@
+# Build the docker image used to run the scripts
+# to continuously update our docker files.
+#
+# The context for this docker file should be the root of the kubeflow/testing repository.
 FROM ubuntu:18.04
 
 RUN apt-get update -y && \
@@ -22,22 +26,28 @@ RUN export KUSTOMIZE_VERSION=3.2.0 && \
     mv kustomize_${KUSTOMIZE_VERSION}_linux_amd64 /usr/local/bin/kustomize && \
     chmod a+x /usr/local/bin/kustomize
 
-
-RUN python -m pip install fire \
-     google-api-python-client \
-     google-cloud-storage \
-     kubernetes \
-     watchdog
-
+COPY apps-cd/requirements.txt /tmp
+RUN python -m pip install \
+     -r /tmp/requirements.txt
 
 # Create go symlinks
 RUN ln -sf /usr/local/go/bin/go /usr/local/bin && \
     ln -sf /usr/local/go/bin/gofmt /usr/local/bin && \
     ln -sf /usr/local/go/bin/godoc /usr/local/bin
 
 RUN mkdir -p /app
-COPY update_launcher.py /app
-COPY run_with_auto_restart.py /app
+
+RUN cd /app && \
+    mkdir -p /app/src/kubeflow && \
+    cd /app/src/kubeflow && \
+    git clone  https://github.com/kubeflow/code-intelligence code-intelligence && \
+    cd code-intelligence && \
+    git checkout db1230d
+
+COPY apps-cd/run_with_auto_restart.py /app
+COPY py /app/src/kubeflow/testing/py
+
+ENV PYTHONPATH /app/src/kubeflow/testing/py:/app/src/kubeflow/code-intelligence/py:$PYTHONPATH
 
 # See(https://github.com/tektoncd/pipeline/issues/1271): Tekton will put ssh
 # credentials in /tekton/home. We can't change the home directory

apps-cd/README.md

@@ -43,13 +43,7 @@ and open PRs to update Kubeflow kustomize manifests to use the newly built image
 
  * The kubeflow-bot GitHub account is used to create the PRs
 
- * Continuous building is achieved by running `update_launcher.py` in a Kubernetes deployment
-
-   * This script periodically fetches the latest code in `kubeflow/testing` to pick up any changes
-     to the code or config
-
-   * It then launches `update_kf_apps.py` to create Tekton PipelineRuns for any applications that
-     need to be updated.
+ * Continuous building is achieved by running `update_kf_apps.py` in a Kubernetes deployment
 
 ### Adding Applications to continuous delivery
 
@@ -150,12 +144,21 @@ This is a Kubeflow cluster (v0.6.2) and we rely on that to configure certain thi
 
 ## Developer Guide
 
-You can use skaffold to build a docker image and auto update the deployment running `update_launcher.py`
+You can use skaffold to build a docker image and auto update the deployment running `update_kf_apps.py`
+
+* The namespace `kf-releasing-dev` is intended for trying out your local changes
+* You can update `apps-cd/pipelines/base/config/app-pipeline.template` to pull in your branch of changes to the CD tooling
+
+  * The template defines a PipelineRun that is used to generate the PipelineRun's to update each application
+  * The template defines a Git resource which points to the kubeflow/testing repository to obtain the scripts to update the manifests
+    * You can change this to point to your fork of kubeflow/testing to test your changes
+* You can also deploy any changes to tekton resources to that namespace before trying out
+  in prod. 
 
 1. Run skaffold
 
    ```
-   skaffold dev -v info --cleanup=false --trigger=polling
+   skaffold dev -p dev -v info --cleanup=false --trigger=polling
    ```
 
 1. During development you can take advantage of skaffold's continuous file sync mode to update

apps-cd/applications.yaml

@@ -119,26 +119,3 @@ versions:
               value: v1.0-branch
             - name: url
               value: [email protected]:kubeflow/manifests.git
-  # Define a v0-8 release
-  # This is primarily so we can test that the CI/CD infrastructure is workin 
-  # with release branches. We don't plan on actually releasing 0.8
-  - name: v0-8
-    # A tag to prefix image names with
-    tag: "v0.8.0"
-    repos:
-      - name: kubeflow
-        resourceSpec:    
-          type: git
-          params:
-            - name: revision
-              value: v0.8-branch
-            - name: url
-              value: [email protected]:kubeflow/kubeflow.git
-      - name: manifests
-        resourceSpec:    
-          type: git
-          params:
-            - name: revision
-              value: v0.8-branch
-            - name: url
-              value: [email protected]:kubeflow/manifests.git

apps-cd/runs/app-pipeline.template.yaml → apps-cd/pipelines/base/config/app-pipeline.template.yaml

@@ -45,9 +45,13 @@ spec:
         # https://github.com/kubeflow/testing/pull/572
         # is merged
         - name: revision
-          value: master
+          value: cicd_delete_old_prs
         - name: url
-          value: [email protected]:kubeflow/testing.git
+          value: [email protected]:jlewi/testing.git
+        #- name: revision
+        #  value: master
+        #- name: url
+        #  value: [email protected]:kubeflow/testing.git
   # The image we want to build
   - name:  image
     resourceSpec:

apps-cd/pipelines/base/deployment.yaml

@@ -17,41 +17,47 @@ spec:
       - name: app
         image: gcr.io/kubeflow-releasing/update_kf_apps
         command:
-        # Uncomment the following lines when running with skaffold 
-        # and you want to use skaffold's auto-sync feature wto pick up changes
-        # to update_launcher.
-        # Begin skaffold
         - python
-        - run_with_auto_restart.py
-        - --dir=/app
-        - --
-        # End skaffold
-        #
-        - python
-        - update_launcher.py
-        - run
-        # repo_dir is the directory where kubeflow testing should be checked out to
-        - --repo_dir=/launcher_src/kubeflow/testing
-        # repo is the code to check out to get the configuration as well as code for
-        # launching the applications
-        #
-        # To test changes before they are checked in you can set this to the repo containing your PR
-        # e.g. - --repo=https://github.com/jlewi/testing.git?ref=cicd_0.8
-        - --repo=https://github.com/kubeflow/testing.git                
+        - -m
+        - kubeflow.testing.cd.update_kf_apps
+        - sync
         # Extra arguments to be passed to update_kf_apps.py
-        - --namespace=kf-releasing 
-        - --config=/launcher_src/kubeflow/testing/apps-cd/applications.yaml 
+        - --namespace=kf-releasing
+        # TODO(jlewi): Should we just put this in a config map as well and then check in into source control?
+        # Would a tool like Anthos CM be smart enough to detect changes in that file and trigger
+        # updates with kustomize?
+        - --config=https://raw.githubusercontent.com/kubeflow/testing/master/apps-cd/applications.yaml
         - --output_dir=/tmp/runs 
         - --src_dir=/src 
-        - --template=/launcher_src/kubeflow/testing/apps-cd/runs/app-pipeline.template.yaml 
+        - --template=/app/config/app-pipeline.template.yaml
         env:
+        # TODO(jlewi): We should stop using a GITHUB_TOKEN and switch to using
+        # the github APP. Is this only used by the Hub CLI?
         - name: GITHUB_TOKEN
           valueFrom:
             secretKeyRef:
               name: github-token
               key: github_token
+        - name: GITHUB_APP_PEM_KEY
+          value: /var/secrets/github/kubeflow-auto-bot.2020-01-24.private-key.pem
         resources:
           requests:
             cpu: 4
             memory: 8Gi
         workingDir: /app
+        volumeMounts:
+        - name: github-app
+          mountPath: /var/secrets/github
+        volumeMounts:
+        - name: pipelinerun-template
+          mountPath: /app/config
+      volumes:
+      # Secret containing the PEM key for the GitHub app used to close PRs.
+      - name: github-app
+        secret:
+          secretName: github-app-pem
+      - name: pipelinerun-template
+        configMap:
+          # Kustomize will automatically replace the name with the unique name given
+          # to the configmap based on the config contents.
+          name: pipelinerun-template

apps-cd/pipelines/base/kustomization.yaml

@@ -11,3 +11,9 @@ resources:
 - task.yaml
 - pipeline.yaml
 namespace: kf-releasing
+# Create a configMap containing the template for the pipeline run
+configMapGenerator:
+- name: pipelinerun-template
+  files:
+    # key will be name of the file
+    - ./config/app-pipeline.template.yaml

apps-cd/pipelines/base/task.yaml

@@ -37,7 +37,11 @@ spec:
     - --digest-file=/workspace/image-digest    
     env:
     - name: GOOGLE_APPLICATION_CREDENTIALS
-      value: /secret/user-gcp-sa.json     
+      value: /secret/user-gcp-sa.json
+    resources:
+      requests:
+        cpu: 7
+        memory: 16Gi
     volumeMounts:
     - mountPath: /secret
       name: gcp-credentials  
@@ -88,8 +92,8 @@ spec:
     - -m
     - kubeflow.testing.cd.create_manifests_pr
     - apply 
-    - --image_url=${IMAGE_URL}
-    - --src_image_url=${SRC_IMAGE_URL} 
+    - --image_url=$(inputs.resources.image.url)
+    - --src_image_url=$(inputs.params.src_image_url)
     - --manifests_dir=/workspace/$(inputs.resources.manifests.name)/$(inputs.params.path_to_manifests_dir)
     - --manifests_base=$(inputs.resources.manifests.revision)
     env:
@@ -102,14 +106,10 @@ spec:
         secretKeyRef:
           name: github-token
           key: github_token
-    - name: SRC_IMAGE_URL          
-      value: $(inputs.params.src_image_url)
-    - name: IMAGE_URL          
-      value: $(inputs.resources.image.url)
-    - name: MANIFESTS_REPO_REVISION
-      value: $(inputs.resources.manifests.revision)
-    - name: PATH_TO_MANIFESTS_DIR
-      value: $(inputs.params.path_to_manifests_dir)
+    resources:
+      requests:
+        cpu: 4
+        memory: 4Gi
     volumeMounts:
     - mountPath: /secret
       name: gcp-credentials

apps-cd/pipelines/overlays/dev/deployment.yaml

@@ -0,0 +1,33 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: server
+spec:
+  template:
+    spec:
+      containers:
+      - name: app
+        image: gcr.io/kubeflow-releasing/update_kf_apps
+        command:
+        # In the dev environment we run with auto_restart for compatibility with
+        # skaffold
+        - python
+        - run_with_auto_restart.py
+        - --dir=/app
+        - --
+        - python
+        - -m
+        - kubeflow.testing.cd.update_kf_apps
+        - sync
+        # repo is the code to check out to get the configuration as well as code for
+        # launching the applications
+        #
+        # To test changes before they are checked in you can set this to the repo containing your PR
+        # e.g. 
+        # Extra arguments to be passed to update_kf_apps.py
+        - --namespace=kf-releasing-dev
+        - --config=https://raw.githubusercontent.com/jlewi/testing/cicd_delete_old_prs/apps-cd/applications.yaml
+        - --output_dir=/tmp/runs 
+        - --src_dir=/src 
+        - --template=/app/config/app-pipeline.template.yaml
+

apps-cd/pipelines/overlays/dev/kustomization.yaml

@@ -0,0 +1,9 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+bases:
+- ../../base
+commonLabels:
+  environment: dev
+namespace: kf-releasing-dev
+patchesStrategicMerge:
+- deployment.yaml

apps-cd/pipelines/overlays/prod/kustomization.yaml

@@ -0,0 +1,7 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+bases:
+- ../../base
+commonLabels:
+  environment: prod
+namespace: kf-releasing

apps-cd/requirements.txt

@@ -0,0 +1,20 @@
+-i https://pypi.org/simple
+asn1crypto==0.24.0
+cffi==1.13.2
+fire==0.2.1
+github3.py==1.3.0
+google-api-core==1.14.2
+google-api-python-client==1.7.10
+google-auth==1.6.3
+google-auth-httplib2==0.0.3
+google-cloud-core==1.0.3
+google-cloud-storage==1.17.0
+json-log-formatter==0.2.0
+jwcrypto==0.6.0
+kubernetes==9.0.0
+pyjwt==1.7.1
+requests==2.22.0
+requests-oauthlib==1.2.0
+requests-toolbelt==0.9.1
+tqdm==4.42.0
+watchdog==0.9.0