-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tarfetch: make it a button #11
Open
TimothyLoyer
wants to merge
15
commits into
main
Choose a base branch
from
tarfetch--make-it-a-button
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+269
−78
Open
Changes from all commits
Commits
Show all changes
15 commits
Select commit
Hold shift + click to select a range
937a6d6
format Tiltfile with buildifier
TimothyLoyer 1940a6e
update docstring, removing not-so-starlark typing
TimothyLoyer 7efa8ae
update arguments
TimothyLoyer a9722b8
add deprecation handling for switch to button
TimothyLoyer 50778f4
switch from resource to button
TimothyLoyer e9ed999
update messaging
TimothyLoyer f96d07e
add tarfetch script
TimothyLoyer 2a4c4a4
update readme
TimothyLoyer 5d827a7
remove -i from kubectl exec command
TimothyLoyer c99fad1
update readme
TimothyLoyer 6e4858a
add tests - start with test.sh
TimothyLoyer a906d6a
readme: make quotes consistent
TimothyLoyer 1674ce2
tiltfile: fix how arguments are passed to script
TimothyLoyer 9f583a9
tiltfile: add ability to change button text (i.e. Sync Migrations)
TimothyLoyer 09378ed
tiltfile: fix code that broke when destination didn't exist
TimothyLoyer File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,48 +1,59 @@ | ||
# Tarfetch | ||
|
||
This extension leverages `kubectl exec` and `tar` to archive files in a Kubernetes container and then extract them to a destination on the local filesystem. | ||
_**Made with the file-freshening properties of tar!**_ | ||
|
||
## Overview | ||
This extension leverages `kubectl exec` and `tar` to archive files in a Kubernetes container and extract them to a destination on the local filesystem. | ||
|
||
Tarfetch provides a local resource method (`tarfetch`) to perform manually triggered reverse file synchronization (i.e. from pod container to local filesystem). | ||
```mermaid | ||
flowchart LR | ||
|
||
Tarfetch resources can be triggered manually either via the Tilt Web UI, or via CLI using `tilt trigger <resource name>`. | ||
kubectl[kubectl exec] --> tar | ||
archive -. pipe .-> untar | ||
untar --> dest([Destination]) | ||
|
||
## Usage | ||
subgraph Container | ||
source([Source Files]) ==> tar | ||
tar ==> archive(((Archived Files))) | ||
end | ||
``` | ||
|
||
Tarfetch's only requirement is that both the local machine and container have `tar` installed. This is typically a given ([yes, even on Windows](https://docs.microsoft.com/en-us/virtualization/community/team-blog/2017/20171219-tar-and-curl-come-to-windows)). | ||
|
||
Import Tarfetch with the following in your Tiltfile: | ||
``` | ||
load('ext://tarfetch', 'tarfetch') | ||
## Example | ||
|
||
Create a local resource called "tarfetch-app" which connects to the first pod of "deploy/frontend" (and the default container) and syncs the contents of "/app/" to local directory "./frontend" while ignoring all directories named "node_modules": | ||
|
||
```starlark | ||
# Import extension | ||
load("ext://tarfetch", "tarfetch") | ||
|
||
# Setup tarfetch to attach the button to a Tilt resource | ||
tarfetch( | ||
"tilt-app-resource", | ||
"deployments/frontend", | ||
"/app/", | ||
"./frontend", | ||
ignore=["node_modules"] | ||
) | ||
``` | ||
|
||
## Usage | ||
|
||
A `tarfetch` resource can be created with the following parameters: | ||
|
||
Required parameters: | ||
* **name (str)**: a name for the local resource | ||
* **k8s_object (str)**: a Kubernetes object identifier (e.g. `deploy/my-deploy`, `job/my-job`, or a pod ID) that Tilt can use to select a pod. As per the behavior of `kubectl exec`, we will act on the first pod of the specified object, using the first container by default | ||
### Required parameters | ||
|
||
* **tilt_resource (str)**: name of Tilt resource to bind button to | ||
* **k8s_resource (str)**: a Kubernetes object identifier (e.g. `deploy/my-deploy`, `job/my-job`, or a pod ID) that Tilt can use to select a pod. As per the behavior of `kubectl exec`, we will act on the first pod of the specified object, using the first container by default | ||
* **src_dir (str)**: directory *in the remote container* to sync from. Any `paths`, if specified, should be relative to this dir. This path *must* be a directory and must contain a trailing slash (e.g. `/app/` is acceptable; `/app` is not) | ||
|
||
Optional parameters: | ||
* **target_dir (str, optional)**: directory *on the local filesystem* to sync to. Defaults to `'.'` | ||
### Optional parameters | ||
|
||
* **target_dir (str, optional)**: directory *on the local filesystem* to sync to. Defaults to `"."` | ||
* **namespace (str, optiona)**: namespace of the desired `k8s_object`, if not `default`. | ||
* **container (str, optional)**: name of the container to sync from (by default, the first container) | ||
* **ignore (List[str], optional)**: patterns to ignore when syncing, [see `tar --exclude` documentation for details on supported patterns](https://www.gnu.org/software/tar/manual/html_node/exclude.html). | ||
* **keep_newer (bool, optional)**: prevents files overwrites when the destination file is newer. Default is true. | ||
* **verbose (bool, optional)**: if true, shows tar extract activity. | ||
* **labels (Union[str, List[str]], optional)**: used to group resources in the Web UI. | ||
|
||
### Example invocation | ||
|
||
Create a local resource called "tarfetch-app" which connects to the first pod of "deploy/frontend" (and the default container) and syncs the contents of "/app/" to local directory "./frontend" while ignoring all directories named "node_modules": | ||
|
||
```python | ||
tarfetch( | ||
'tarfetch-app', | ||
'deployments/frontend', | ||
'/app/', | ||
'./frontend', | ||
ignore=["node_modules"] | ||
) | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
#!/usr/bin/env sh | ||
|
||
set -eu | ||
|
||
PACK_ARGS="--directory=${TARFETCH_SRC_DIR}" | ||
[ -n "$TARFETCH_EXCLUDE" ] && PACK_ARGS="${PACK_ARGS} ${TARFETCH_EXCLUDE}" | ||
|
||
UNPACK_ARGS="--directory=${TARFETCH_TARGET_DIR}" | ||
[ "$TARFETCH_KEEP_NEWER" = "true" ] && UNPACK_ARGS="${UNPACK_ARGS} --keep-newer-files" | ||
|
||
if [ "$TARFETCH_VERBOSE" = "true" ]; then | ||
UNPACK_ARGS="${UNPACK_ARGS} --verbose" | ||
set -x | ||
fi | ||
|
||
pack() { | ||
kubectl exec -n "$TARFETCH_NAMESPACE" "$TARFETCH_RESOURCE_NAME" -- \ | ||
tar -c -f - $PACK_ARGS . | ||
} | ||
|
||
unpack() { | ||
tar -x -f - $UNPACK_ARGS | ||
} | ||
|
||
pack | unpack | ||
|
||
echo '[tarfetch] Done: Sync from container has finished.' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
load("../Tiltfile", "tarfetch") | ||
|
||
k8s_yaml("k8s/pod-tarfetch.yaml") | ||
|
||
tarfetch( | ||
"tarfetch-example", | ||
"pods/tarfetch-example", | ||
"/app/", | ||
"./files/", | ||
ignore = [ | ||
"**/dont", | ||
"dont.*", | ||
] | ||
) | ||
|
||
local_resource( | ||
"tarfetch-tests", | ||
cmd = ["./test-tarfetch.sh"], | ||
resource_deps = ["tarfetch-example"] | ||
) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
apiVersion: v1 | ||
kind: Pod | ||
metadata: | ||
name: tarfetch-example | ||
spec: | ||
terminationGracePeriodSeconds: 1 | ||
containers: | ||
- name: tarfetch-example | ||
image: busybox:latest | ||
imagePullPolicy: IfNotPresent | ||
workingDir: /app | ||
command: | ||
- sh | ||
- -euc | ||
args: | ||
- | | ||
mkdir do | ||
mkdir dont | ||
|
||
touch do.sync | ||
touch dont.sync | ||
touch do/do.sync | ||
touch do/dont.sync | ||
touch dont/do.sync | ||
touch dont/dont.sync | ||
|
||
sleep infinity |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
#!/usr/bin/env bash | ||
set -eu | ||
|
||
successes=0 | ||
failures=0 | ||
|
||
print_test() { | ||
message="$1" | ||
shift | ||
|
||
if [ $@ ]; then | ||
state="\033[32mPASSED\033[0m" | ||
successes=$((successes + 1)) | ||
else | ||
state="\033[31mFAILED\033[0m" | ||
failures=$((failures + 1)) | ||
fi | ||
|
||
echo -e "${state} - ${message}" | ||
} | ||
|
||
./trigger.sh btn-tarfetch-tarfetch-example | ||
sleep 2 | ||
|
||
echo | ||
echo "Test results:" | ||
|
||
print_test "Test files/ exists" -d files/ | ||
print_test "Test do.sync exists" -f files/do.sync | ||
print_test "Test dont.sync does not exist" ! -f files/dont.sync | ||
print_test "Test do/ exists" -d files/do/ | ||
print_test "Test do/do.sync exists" -f files/do/do.sync | ||
print_test "Test do/dont.sync does not exist" ! -f files/do/dont.sync | ||
print_test "Test dont/ does not exist" ! -d files/dont/ | ||
|
||
echo | ||
echo "Ran $((failures + successes)) tests" | ||
if [ "$failures" = "0" ]; then | ||
echo -e "\033[32mOK\033[0m" | ||
else | ||
echo -e "\033[31mFAILED (failures=${failures}, successes=${successes})\033[0m" | ||
fi | ||
echo |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
#!/usr/bin/env bash | ||
set -eu | ||
|
||
cleanup() { | ||
find ./files -type f -name '*.do' -delete | ||
find ./files -type f -name '*.dont' -delete | ||
} | ||
|
||
cd "$(dirname "$0")" | ||
|
||
echo "Preparing sync destination..." | ||
if [ -d ./files ]; then | ||
cleanup | ||
else | ||
mkdir ./files | ||
fi | ||
|
||
tilt ci | ||
tilt down | ||
|
||
echo "Cleaning up test files..." | ||
cleanup | ||
rm -r ./files | ||
|
||
echo "Done" |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you include a section here explaining why someone might want to use this tar-based method of getting files out of a container instead of using
kubectl cp
?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll explain it to you, and then try to condense an explanation for the readme.
Here is what
kubectl cp --help
says:These more advanced examples are what tarfetch is doing - allowing a bit more flexibility than
kubectl cp
and puts it in a nice UI button.I mentioned flexibility... in the less advanced examples for
kubectl cp
, it can only work when you know the exact name of the pod (odd since you can use a pattern likedeploy/<deployment name>
when using thekubectl exec
examples). This means we can create a sync button which points at a deployment as a sync target.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe something like...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That looks good to me (minus the "Kubectp" typo).