Tarfetch: make it a button

tarfetch/README.md

@@ -1,48 +1,59 @@
 # Tarfetch
 
-This extension leverages `kubectl exec` and `tar` to archive files in a Kubernetes container and then extract them to a destination on the local filesystem.
+_**Made with the file-freshening properties of tar!**_
 
-## Overview
+This extension leverages `kubectl exec` and `tar` to archive files in a Kubernetes container and extract them to a destination on the local filesystem.
 
-Tarfetch provides a local resource method (`tarfetch`) to perform manually triggered reverse file synchronization (i.e. from pod container to local filesystem).
+```mermaid
+flowchart LR
 
-Tarfetch resources can be triggered manually either via the Tilt Web UI, or via CLI using `tilt trigger <resource name>`.
+    kubectl[kubectl exec] --> tar
+    archive -. pipe .-> untar
+    untar --> dest([Destination])
 
-## Usage
+    subgraph Container
+        source([Source Files]) ==> tar
+        tar ==> archive(((Archived Files)))
+    end
+```
 
 Tarfetch's only requirement is that both the local machine and container have `tar` installed. This is typically a given ([yes, even on Windows](https://docs.microsoft.com/en-us/virtualization/community/team-blog/2017/20171219-tar-and-curl-come-to-windows)).
 
-Import Tarfetch with the following in your Tiltfile:
-```
-load('ext://tarfetch', 'tarfetch')
+## Example
+
+Create a local resource called "tarfetch-app" which connects to the first pod of "deploy/frontend" (and the default container) and syncs the contents of "/app/" to local directory "./frontend" while ignoring all directories named "node_modules":
+
+```starlark
+# Import extension
+load("ext://tarfetch", "tarfetch")
+
+# Setup tarfetch to attach the button to a Tilt resource
+tarfetch(
+   "tilt-app-resource", 
+   "deployments/frontend",
+   "/app/",
+   "./frontend",
+   ignore=["node_modules"]
+)
 ```
 
+## Usage
+
 A `tarfetch` resource can be created with the following parameters:
 
-Required parameters:
-* **name (str)**: a name for the local resource
-* **k8s_object (str)**: a Kubernetes object identifier (e.g. `deploy/my-deploy`, `job/my-job`, or a pod ID) that Tilt can use to select a pod. As per the behavior of `kubectl exec`, we will act on the first pod of the specified object, using the first container by default
+### Required parameters
+
+* **tilt_resource (str)**: name of Tilt resource to bind button to
+* **k8s_resource (str)**: a Kubernetes object identifier (e.g. `deploy/my-deploy`, `job/my-job`, or a pod ID) that Tilt can use to select a pod. As per the behavior of `kubectl exec`, we will act on the first pod of the specified object, using the first container by default
 * **src_dir (str)**: directory *in the remote container* to sync from. Any `paths`, if specified, should be relative to this dir. This path *must* be a directory and must contain a trailing slash (e.g. `/app/` is acceptable; `/app` is not)
 
-Optional parameters:
-* **target_dir (str, optional)**: directory *on the local filesystem* to sync to. Defaults to `'.'`
+### Optional parameters
+
+* **target_dir (str, optional)**: directory *on the local filesystem* to sync to. Defaults to `"."`
 * **namespace (str, optiona)**: namespace of the desired `k8s_object`, if not `default`.
 * **container (str, optional)**: name of the container to sync from (by default, the first container)
 * **ignore (List[str], optional)**: patterns to ignore when syncing, [see `tar --exclude` documentation for details on supported patterns](https://www.gnu.org/software/tar/manual/html_node/exclude.html).
 * **keep_newer (bool, optional)**: prevents files overwrites when the destination file is newer. Default is true.
 * **verbose (bool, optional)**: if true, shows tar extract activity.
-* **labels (Union[str, List[str]], optional)**: used to group resources in the Web UI.
 
-### Example invocation
 
-Create a local resource called "tarfetch-app" which connects to the first pod of "deploy/frontend" (and the default container) and syncs the contents of "/app/" to local directory "./frontend" while ignoring all directories named "node_modules":
-
-```python
-tarfetch(
-   'tarfetch-app', 
-   'deployments/frontend',
-   '/app/',
-   './frontend',
-   ignore=["node_modules"]
-)
- ```

tarfetch/Tiltfile

@@ -1,5 +1,4 @@
-# -*- mode: Python -*-
-
+TARFETCH_SCRIPT = os.path.join(os.getcwd(), "scripts", "tarfetch.sh")
 DEFAULT_EXCLUDES = [
     ".git",
     ".gitignore",
@@ -11,45 +10,57 @@ DEFAULT_EXCLUDES = [
 ]
 
 def tarfetch(
-    name,
-    k8s_object,
+    tilt_resource,
+    k8s_resource,
     src_dir,
-    target_dir=".",
-    namespace="default",
-    container="",
-    ignore=None,
-    keep_newer=True,
-    verbose=False,
-    labels=tuple()
+    target_dir = ".",
+    namespace = "default",
+    container = "",
+    ignore = None,
+    keep_newer = True,
+    verbose = False,
+    labels = None,
+    button_text = None,
 ):
     """
-    Create a local resource that will (via rsync) sync the specified files
-    from the specified k8s object to the local filesystem.
+    Create a sync button on the specified Tilt resource, which will pull files from
+    a Kubernetes container onto the local filesystem.
 
-    :param name (str): name of the created local resource.
-    :param k8s_object (str): a Kubernetes object identifier (e.g. deploy/my-deploy,
+    :param tilt_resource: name of Tilt resource to bind button to.
+    :param k8s_resource: a Kubernetes object identifier (e.g. deploy/my-deploy,
         job/my-job, or a pod ID) that Tilt can use to select a pod. As per the
         behavior of `kubectl exec`, we will act on the first pod of the specified
         object, using the first container by default.
-    :param src_dir (str): directory IN THE KUBERNETES CONTAINER to sync from. Any
+    :param src_dir: directory IN THE KUBERNETES CONTAINER to sync from. Any
         paths specified, if relative, should be relative to this dir.
-    :param target_dir (str, optional): directory ON THE LOCAL FS to sync to. Defaults to '.'
-    :param namespace (str, optional): namespace of the desired k8s_object, if not `default`.
-    :param container (str, optional): name of the container to sync from (by default,
+    :param target_dir: directory ON THE LOCAL FS to sync to. Defaults to '.'
+    :param namespace: namespace of the desired k8s_object, if not `default`.
+    :param container: name of the container to sync from (by default,
         the first container)
-    :param ignore (List[str], optional): patterns to ignore when syncing, see
+    :param ignore: patterns to ignore when syncing, see
         `tar --exclude` documentation for details on supported patterns.
-    :param keep_newer (bool, optional): prevents files overwrites when the destination
+    :param keep_newer: prevents files overwrites when the destination
         file is newer. Default is true.
-    :param verbose (bool, optional): if true, shows tar extract activity.
-    :return:
+    :param verbose: if true, shows tar extract activity.
+    :param labels: deprecated argument from when tarfetch was a resource rather than
+        a UI button.
+    :param button_text: provide custom text for button label.
     """
 
+    # Deprecation handling
+    if tilt_resource.startswith("sync") or tilt_resource.startswith("tarfetch"):
+        warn(
+            "[tarfetch] WARNING: The leading positional argument 'tilt_resource' may "
+            + "have changed in purpose since it was configured for this project. "
+            + "Please update it to reference the Tilt UI resource to bind the sync "
+            + "button to."
+        )
+    if labels:
+        warn("[tarfetch] WARNING: The 'labels' argument has been deprecated and may be removed.")
+
     # Verify inputs
     if not src_dir.endswith("/"):
-        fail(
-            "src_dir must be a directory and have a trailing slash (because of rsync syntax rules)"
-        )
+        fail("[tarfetch] src_dir must be a directory and have a trailing slash")
 
     to_exclude = ignore
     if not ignore:
@@ -62,36 +73,48 @@ def tarfetch(
 
     # bundle container flag with k8s object specifier
     if container:
-        k8s_object = "{obj} -c {container}".format(obj=k8s_object, container=container)
+        k8s_resource = "{obj} -c {container}".format(obj = k8s_resource, container = container)
 
-    destination_path = os.path.realpath(target_dir)
-    if not os.path.exists(destination_path):
-        print("Preparing destination path for reverse sync:")
+    if not os.path.exists(target_dir):
+        print("[tarfetch] Preparing destination path for reverse sync:")
         local(
-            ["mkdir", "-p", destination_path],
-            command_bat="mkdir {} || ver>nul".format(destination_path),
+            ["mkdir", "-p", target_dir],
+            command_bat = "mkdir {} || ver>nul".format(target_dir),
             quiet = True,
         )
 
-    local_resource(
-        name,
-        (
-            "(" +
-            "kubectl exec -i -n {namespace} {k8s_object} -- " +
-            "tar -c -f - --atime-preserve=system --directory={src_dir} {exclude} ." +
-            ") | " +
-            "tar -x -f - {verbose} {keep_newer} --directory={target_dir} && " +
-            "echo Done."
-        ).format(
-            namespace=namespace,
-            k8s_object=k8s_object,
-            exclude=excludes,
-            src_dir=src_dir,
-            target_dir=target_dir,
-            keep_newer="--keep-newer-files" if keep_newer else "",
-            verbose="--verbose" if verbose else "",
+
+    btn_name = "btn-tarfetch-" + tilt_resource
+    v1alpha1.ui_button(
+        name = btn_name,
+        location = {
+            "component_type": "Resource",
+            "component_id": tilt_resource,
+        },
+        text = button_text or "Sync from Container",
+        icon_name = "cloud_sync",
+        annotations = {"tilt.dev/resource": tilt_resource},
+    )
+
+    env = {
+        "namespace": namespace,
+        "resource_name": k8s_resource,
+        "exclude": excludes,
+        "src_dir": src_dir,
+        "target_dir": target_dir,
+        "keep_newer": str(bool(keep_newer)).lower(),
+        "verbose": str(bool(verbose)).lower(),
+    }
+
+    v1alpha1.cmd(
+        name = "cmd-tarfetch-" + tilt_resource,
+        annotations = {
+            "tilt.dev/resource": tilt_resource,
+            "tilt.dev/log-span-id": "cmd:tarfetch:" + tilt_resource,
+        },
+        args = [TARFETCH_SCRIPT],
+        env = ["TARFETCH_%s=%s" % (k.upper(), v) for k, v in env.items()],
+        start_on = v1alpha1.start_on_spec(
+            ui_buttons = [btn_name],
         ),
-        trigger_mode=TRIGGER_MODE_MANUAL,
-        auto_init=False,
-        labels=labels,
     )

tarfetch/scripts/tarfetch.sh

@@ -0,0 +1,27 @@
+#!/usr/bin/env sh
+
+set -eu
+
+PACK_ARGS="--directory=${TARFETCH_SRC_DIR}"
+[ -n "$TARFETCH_EXCLUDE" ] && PACK_ARGS="${PACK_ARGS} ${TARFETCH_EXCLUDE}"
+
+UNPACK_ARGS="--directory=${TARFETCH_TARGET_DIR}"
+[ "$TARFETCH_KEEP_NEWER" = "true" ] && UNPACK_ARGS="${UNPACK_ARGS} --keep-newer-files"
+
+if [ "$TARFETCH_VERBOSE" = "true" ]; then
+  UNPACK_ARGS="${UNPACK_ARGS} --verbose"
+  set -x
+fi
+
+pack() {
+  kubectl exec -n "$TARFETCH_NAMESPACE" "$TARFETCH_RESOURCE_NAME" -- \
+    tar -c -f - $PACK_ARGS .
+}
+
+unpack() {
+  tar -x -f - $UNPACK_ARGS
+}
+
+pack | unpack
+
+echo '[tarfetch] Done: Sync from container has finished.'

tarfetch/test/Tiltfile

@@ -0,0 +1,20 @@
+load("../Tiltfile", "tarfetch")
+
+k8s_yaml("k8s/pod-tarfetch.yaml")
+
+tarfetch(
+   "tarfetch-example",
+   "pods/tarfetch-example",
+   "/app/",
+   "./files/",
+   ignore = [
+       "**/dont",
+      "dont.*",
+   ]
+)
+
+local_resource(
+  "tarfetch-tests",
+  cmd = ["./test-tarfetch.sh"],
+  resource_deps = ["tarfetch-example"]
+)

tarfetch/test/k8s/pod-tarfetch.yaml

@@ -0,0 +1,27 @@
+apiVersion: v1
+kind: Pod
+metadata:
+  name: tarfetch-example
+spec:
+  terminationGracePeriodSeconds: 1
+  containers:
+  - name: tarfetch-example
+    image: busybox:latest
+    imagePullPolicy: IfNotPresent
+    workingDir: /app
+    command:
+    - sh
+    - -euc
+    args:
+    - |
+      mkdir do
+      mkdir dont
+
+      touch do.sync
+      touch dont.sync
+      touch do/do.sync
+      touch do/dont.sync
+      touch dont/do.sync
+      touch dont/dont.sync
+
+      sleep infinity

tarfetch/test/test-tarfetch.sh

@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+set -eu
+
+successes=0
+failures=0
+
+print_test() {
+  message="$1"
+  shift
+
+  if [ $@ ]; then
+    state="\033[32mPASSED\033[0m"
+    successes=$((successes + 1))
+  else
+    state="\033[31mFAILED\033[0m"
+    failures=$((failures + 1))
+  fi
+
+  echo -e "${state} - ${message}"
+}
+
+./trigger.sh btn-tarfetch-tarfetch-example
+sleep 2
+
+echo
+echo "Test results:"
+
+print_test "Test files/ exists" -d files/
+print_test "Test do.sync exists" -f files/do.sync
+print_test "Test dont.sync does not exist" ! -f files/dont.sync
+print_test "Test do/ exists" -d files/do/
+print_test "Test do/do.sync exists" -f files/do/do.sync
+print_test "Test do/dont.sync does not exist" ! -f files/do/dont.sync
+print_test "Test dont/ does not exist" ! -d files/dont/
+
+echo
+echo "Ran $((failures + successes)) tests"
+if [ "$failures" = "0" ]; then
+  echo -e "\033[32mOK\033[0m"
+else
+  echo -e "\033[31mFAILED (failures=${failures}, successes=${successes})\033[0m"
+fi
+echo

tarfetch/test/test.sh

@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+set -eu
+
+cleanup() {
+  find ./files -type f -name '*.do' -delete
+  find ./files -type f -name '*.dont' -delete
+}
+
+cd "$(dirname "$0")"
+
+echo "Preparing sync destination..."
+if [ -d ./files ]; then
+  cleanup
+else
+  mkdir ./files
+fi
+
+tilt ci
+tilt down
+
+echo "Cleaning up test files..."
+cleanup
+rm -r ./files
+
+echo "Done"