Introduce KOCACHE #269

jonjohnsonjr · 2020-12-16T00:29:59Z

Start of #264

Currently, use this by setting KOCACHE=/tmp/ko (or any directory you want). We can default to doing this in the future.

We will cache binaries and metdata under $KOCACHE/bin/<import path>/<platform>:

$ tree /tmp/ko
/tmp/ko                                                                                                                                                                                                                                                                                                                        
`-- bin                                                                                                                                                                                                                                                                                                                        
    `-- github.com                                                                                                                                                                                                                                                                                                             
        `-- google                                                                                                                                                                                                                                                                                                             
            `-- ko                                                                                                                                                                                                                                                                                                             
                `-- linux                                                                                                                                                                                                                                                                                                      
                    |-- amd64                                                                                                                                                                                                                                                                                                  
                    |   |-- buildid-to-diffid                                                                                                                                                                                                                                                                                  
                    |   |-- diffid-to-descriptor                                                                                                                                                                                                                                                                               
                    |   `-- out                                                                                                                                                                                                                                                                                                
                    |-- arm                                                                                                                                                                                                                                                                                                    
                    |   |-- buildid-to-diffid                                                                                                                                                                                                                                                                                  
                    |   |-- diffid-to-descriptor                                                                                                                                                                                                                                                                               
                    |   `-- out                                                                                                                                                                                                                                                                                                
                    |-- arm64                                                                                                                                                                                                                                                                                                  
                    |   |-- buildid-to-diffid                                                                                                                                                                                                                                                                                  
                    |   |-- diffid-to-descriptor                                                                                                                                                                                                                                                                               
                    |   `-- out                                                                                                                                                                                                                                                                                                
                    |-- ppc64le                                                                                                                                                                                                                                                                                                
                    |   |-- buildid-to-diffid                                                                                                                                                                                                                                                                                  
                    |   |-- diffid-to-descriptor                                                                                                                                                                                                                                                                               
                    |   `-- out                                                                                                                                                                                                                                                                                                
                    `-- s390x                                                                                                                                                                                                                                                                                                  
                        |-- buildid-to-diffid                                                                                                                                                                                                                                                                                  
                        |-- diffid-to-descriptor                                                                                                                                                                                                                                                                               
                        `-- out

In buildid-to-diffid, we maintain a map from the binary's buildid to the diffid we produced from it:

$ cat /tmp/ko/bin/github.com/google/ko/linux/amd64/buildid-to-diffid 
{
  "DpZ76uEtG19-5zUMMduu/ZydApuKMqo03IenzJjld/E9kHVxb9IXx9H09h2JH0/vW8Nmsi20Na9bBDsFDYX": "sha256:fb701193c5e0cedb4291fb7308219cc95ec20afe04e24491be1cedf72c3f7a4a",
  "IL1S_KSUq5wIP90ZkKhZ/j-1pixYk9Al_YB-B02oM/QDXIdXeDJc-14xTSVCTB/7wkOkmLWlnCRO42sVTBS": "sha256:d7640b5ef558e77eb69caac1405be6729dde3c7a4c9ccb6ba7fb4ea3812858ca"
}

You can determine the buildid from the binary:

$ go tool buildid /tmp/ko/bin/github.com/google/ko/linux/amd64/out 
IL1S_KSUq5wIP90ZkKhZ/j-1pixYk9Al_YB-B02oM/QDXIdXeDJc-14xTSVCTB/7wkOkmLWlnCRO42sVTBS

Which allows us to skip computing the diffid (tar + sha256) at all.

In diffid-to-descriptor, we maintain a map from diffid to a layer descriptor:

$ cat /tmp/ko/bin/github.com/google/ko/linux/amd64/diffid-to-descriptor 
{
  "sha256:d7640b5ef558e77eb69caac1405be6729dde3c7a4c9ccb6ba7fb4ea3812858ca": {
    "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
    "size": 12162562,
    "digest": "sha256:d4695baddba464a9e730d1d7580ae2aba998ae77980de64b1c4b28a945412519"
  },
  "sha256:fb701193c5e0cedb4291fb7308219cc95ec20afe04e24491be1cedf72c3f7a4a": {
    "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
    "size": 12173723,
    "digest": "sha256:ddc4badd9eacf3b55c9254ca62bd737b3994f8b1c03e99f9873e776889cd2c2b"
  }
}

... which is enough info to HEAD the blob in-registry without having to actually re-gzip it.

This is a separate structure from buildid-to-diffid so that we can decouple diffid-to-descriptor from go builds and reuse it for the kodata layers (or really anything -- docker does very similar stuff here): #262

@mattmoor @imjasonh

Numbers

Warm + cache miss (and fill) + registry miss:

$ time KOCACHE=/tmp/ko KO_DOCKER_REPO=localhost:1338/ko ko publish --platform=all .

real  0m10.501s
user  0m11.848s
sys   0m4.252s

Warm + cache miss (and fill) + registry hits:

$ time KOCACHE=/tmp/ko KO_DOCKER_REPO=localhost:1338/ko ko publish --platform=all .

real  0m9.456s
user  0m11.410s
sys   0m4.165s

Warm + cache hits + registry miss:

$ time KOCACHE=/tmp/ko KO_DOCKER_REPO=localhost:1338/ko ko publish --platform=all .

real  0m5.721s
user  0m6.478s
sys   0m3.758s

Warm + cache hits + registry hits:

$ time KOCACHE=/tmp/ko KO_DOCKER_REPO=localhost:1338/ko ko publish --platform=all .

real  0m4.071s
user  0m5.216s
sys   0m4.358s

What more can we do? Well... about half of this time is actually spent fetching the base image. If we pull that down to a local registry:

$ crane cp gcr.io/distroless/static:nonroot localhost:1338/distroless:nonroot
$ time KO_DEFAULTBASEIMAGE=localhost:1338/distroless:nonroot KOCACHE=/tmp/ko KO_DOCKER_REPO=localhost:1338/ko ko publish --platform=all .

real  0m1.682s
user  0m3.750s
sys   0m2.630s

Starting towards achieving that in #525

jonjohnsonjr · 2020-12-16T00:56:59Z

@mattmoor boilerplate requires /* */ instead of // now?

mattmoor · 2020-12-16T02:22:16Z

@jonjohnsonjr boilerplate requests whatever is in hack/boilerplate/boilerplate.go.txt IIRC

mattmoor

If we were to start making tarball.LayerFromOpener optionally return an estargz layer, what about this would need to change?

pkg/build/gobuild.go

mattmoor · 2020-12-16T02:26:49Z

pkg/build/gobuild.go

+		hasher := md5.New() //nolint: gosec // No strong cryptography needed.
+		hasher.Write([]byte(strings.Join(args, " ") + " " + strings.Join(defaultEnv, " ")))
+
+		tmpDir = filepath.Join(os.TempDir(), "ko", ip, hashInputs(args, defaultEnv))


should you factor in the working directory? What if I have multiple ko workspaces? 🤔

mattmoor · 2020-12-16T02:27:03Z

pkg/build/gobuild.go

+		if os.Getenv("KO_STABLE_OUTPUT") == "" {
+			os.RemoveAll(tmpDir)
+		}


Why cache if it is an error?

Because we don't want to delete the other metadata in that directory, even though the out binary is bad (or never got built). I think I want to rework this directory structure a bit before we merge it, e.g. maybe buildid-to-diffid and diffid-to-descriptor are global and have the same layout as go env GOCACHE.

github-actions · 2021-05-05T01:23:53Z

This Pull Request is stale because it has been open for 90 days with
no activity. It will automatically close after 30 more days of
inactivity. Reopen with /reopen. Mark as fresh by adding the
comment /remove-lifecycle stale.

imjasonh · 2021-05-05T01:38:37Z

/remove-lifecycle stale

I still want this.

jonjohnsonjr · 2021-05-05T16:25:57Z

I don't think we have a bot that actually obeys us. I've just been manually changing the labels.

codecov-commenter · 2021-12-06T17:44:57Z

Codecov Report

Merging #269 (75914d7) into main (5640c33) will decrease coverage by 2.72%.
The diff coverage is 8.72%.

@@            Coverage Diff             @@
##             main     #269      +/-   ##
==========================================
- Coverage   50.60%   47.88%   -2.73%     
==========================================
  Files          41       41              
  Lines        2051     2170     +119     
==========================================
+ Hits         1038     1039       +1     
- Misses        838      955     +117     
- Partials      175      176       +1

Impacted Files	Coverage Δ
pkg/build/layer.go	`0.00% <0.00%> (ø)`
pkg/build/gobuild.go	`46.40% <9.77%> (-10.05%)`	⬇️
pkg/commands/commands.go	`72.72% <0.00%> (-2.28%)`	⬇️
pkg/commands/completion.go

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5640c33...75914d7. Read the comment docs.

Cache binaries under $KOCACHE/<import path>/<platform> Cache metdata mapping buildid to diffid and diffid to descriptor.

jonjohnsonjr · 2021-12-07T00:17:38Z

Want to take a look and fiddle with it?

pkg/build/gobuild.go

jonjohnsonjr · 2021-12-07T17:21:46Z

I'm tempted to namespace stuff a little bit like $KOCACHE/bin/<import path> instead of just $KOCACHE/<import path>.

Then we could cache base image metadata under $KOCACHE/img/<repo> or something.

pkg/build/gobuild.go

This makes things a little cleaner by having a single place that calls buildLayer and passing a thunk down into the cache logic to call that on a cache miss. Also, remove the debug logging to make the code easier to follow (if you need to recompile anyway, it's easy enough to add log lines).

mattmoor

🚀

jonjohnsonjr marked this pull request as draft December 16, 2020 01:04

mattmoor reviewed Dec 16, 2020

View reviewed changes

jonjohnsonjr mentioned this pull request Jan 11, 2021

FR: Annotate images with build information #221

Open

Base automatically changed from master to main February 3, 2021 19:56

imjasonh mentioned this pull request Feb 19, 2021

Build go binaries in parallel #314

Open

github-actions bot added the lifecycle/stale label May 5, 2021

jonjohnsonjr added lifecycle/frozen and removed lifecycle/stale labels May 5, 2021

imjasonh mentioned this pull request Aug 9, 2021

RFC: deprecate and remove Watch Mode #412

Closed

jonjohnsonjr force-pushed the ko-faster branch from 723be95 to 3f433e7 Compare December 6, 2021 17:40

google-cla bot added the cla: yes label Dec 6, 2021

jonjohnsonjr force-pushed the ko-faster branch 4 times, most recently from f6da708 to 4ea7b57 Compare December 7, 2021 00:01

Introduce KOCACHE

021fa01

Cache binaries under $KOCACHE/<import path>/<platform> Cache metdata mapping buildid to diffid and diffid to descriptor.

jonjohnsonjr force-pushed the ko-faster branch from 4ea7b57 to 021fa01 Compare December 7, 2021 00:12

jonjohnsonjr marked this pull request as ready for review December 7, 2021 00:17

jonjohnsonjr assigned imjasonh and mattmoor Dec 7, 2021

mattmoor reviewed Dec 7, 2021

View reviewed changes

pkg/build/gobuild.go Show resolved Hide resolved

mattmoor reviewed Dec 7, 2021

View reviewed changes

pkg/build/gobuild.go Outdated Show resolved Hide resolved

jonjohnsonjr changed the title ~~POC: KO_META_CACHE layer info~~ Introduce KOCACHE Dec 7, 2021

Point TODO at issue

75914d7

mattmoor previously approved these changes Dec 7, 2021

View reviewed changes

imjasonh previously approved these changes Dec 7, 2021

View reviewed changes

pkg/build/gobuild.go Outdated Show resolved Hide resolved

imjasonh reviewed Dec 7, 2021

View reviewed changes

pkg/build/gobuild.go Outdated Show resolved Hide resolved

jonjohnsonjr dismissed stale reviews from imjasonh and mattmoor via 75e0a4c December 7, 2021 18:15

jonjohnsonjr force-pushed the ko-faster branch 2 times, most recently from a4bb483 to 42179d5 Compare December 7, 2021 18:19

jonjohnsonjr force-pushed the ko-faster branch from 42179d5 to 8098926 Compare December 7, 2021 18:22

Move cached output to $KOCACHE/bin

9a9ece2

jonjohnsonjr force-pushed the ko-faster branch from f4f5608 to 9a9ece2 Compare December 7, 2021 18:39

imjasonh approved these changes Dec 8, 2021

View reviewed changes

mattmoor approved these changes Dec 8, 2021

View reviewed changes

imjasonh merged commit 54cddcc into ko-build:main Dec 8, 2021

This was referenced Aug 29, 2022

add a new section to README about KOCACHE capability #806

Closed

[Feature] use KOCACHE environment variable to cache layers to speed up the build process kyverno/kyverno#4439

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce KOCACHE #269

Introduce KOCACHE #269

jonjohnsonjr commented Dec 16, 2020 •

edited

Loading

jonjohnsonjr commented Dec 16, 2020

mattmoor commented Dec 16, 2020

mattmoor left a comment

mattmoor Dec 16, 2020

mattmoor Dec 16, 2020

jonjohnsonjr Dec 16, 2020 •

edited

Loading

github-actions bot commented May 5, 2021

imjasonh commented May 5, 2021

jonjohnsonjr commented May 5, 2021

codecov-commenter commented Dec 6, 2021 •

edited

Loading

jonjohnsonjr commented Dec 7, 2021

jonjohnsonjr commented Dec 7, 2021 •

edited

Loading

mattmoor left a comment

Introduce KOCACHE #269

Introduce KOCACHE #269

Conversation

jonjohnsonjr commented Dec 16, 2020 • edited Loading

Numbers

jonjohnsonjr commented Dec 16, 2020

mattmoor commented Dec 16, 2020

mattmoor left a comment

Choose a reason for hiding this comment

mattmoor Dec 16, 2020

Choose a reason for hiding this comment

mattmoor Dec 16, 2020

Choose a reason for hiding this comment

jonjohnsonjr Dec 16, 2020 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented May 5, 2021

imjasonh commented May 5, 2021

jonjohnsonjr commented May 5, 2021

codecov-commenter commented Dec 6, 2021 • edited Loading

Codecov Report

jonjohnsonjr commented Dec 7, 2021

jonjohnsonjr commented Dec 7, 2021 • edited Loading

mattmoor left a comment

Choose a reason for hiding this comment

jonjohnsonjr commented Dec 16, 2020 •

edited

Loading

jonjohnsonjr Dec 16, 2020 •

edited

Loading

codecov-commenter commented Dec 6, 2021 •

edited

Loading

jonjohnsonjr commented Dec 7, 2021 •

edited

Loading