Implement Bazel Remote Caching [Tracking Issue]

[BenTheElder]

Bazel 0.10.0 is now in use in test-infra and kubernetes (release-1.10/master). This release contains some nice improvements to the HTTP remote caching system, we should leverage this instead of our existing "use persistent storage for the local cache".

Why?

• using a remote cache means the cache is global so all jobs can share it
• the remote cache is (mostly) content addressed and designed for sharing, the local cache is not so much
• we can't have a broken node with a bad cache to hunt down if we don't put the cache on the node

Why not?

• it's another thing to deploy and keep an eye on
• bugs / invalid caching can break builds
  • More specifically see remote/correctness: Bazel doesn't track system libraries. bazelbuild/bazel#4558
  • NOTE: we've found multiple work arounds for this and implemented one, see below

Action Items

• configure bazel for remote caching and test this by building and testing test-infra, kubernetes, etc.
• automate configuration (ref: point test-infra bazel canary at the cache #6661, set local fallback and correct domain #6673, document and improve bazel caching args, apply to more canary jobs #6682, fix create_bazel_cache_rcs.sh #6685)
  • NOTE: we're working around the invalid caching issue by auto configuring jobs to point to a (per-repo, per-host-tools-versions (hashed)) cache, so they simply use a different cache
• evaluate caching backends (some discussion here: use per WORKSPACE / host tools keyed bazel cache(s) #6767)
  • GCS is nice and definitely something to consider, but having the cache in-cluster should be faster and we can control the cache size / costs / eviction better.
  • Hazelcast is a fancy distributed cache, but much more complex and comes with JVM baggage
  • nginx with webdav or apache httpd with webdav works, but in particular doesn't handle eviction, stats etc.
  • bazel-remote is great, but we want to host one server for multiple "caches" so we can split up repos etc. to be defensive about incorrect caching
  • Our own minimal implementation: easy enough, the protocol is essentially just HTTP PUT/GET of blobs
• Improve our implementation
  • implement an eviction strategy (bazel remote cache: add eviction #6812)
  • document (move/rename experiment/nursery to greenhouse, document, enable for test-infra #6879)
  • rename from nursery to greenhouse to be less confusing :-) (move/rename experiment/nursery to greenhouse, document, enable for test-infra #6879)
  • add metrics
• set up monitoring / dashboard
• roll out to "real" jobs / integrate with our images [WIP (test-infra)]
  • integrate into our images (move bazel cache rc creation into images #6859)
  • enable for test-infra (move/rename experiment/nursery to greenhouse, document, enable for test-infra #6879)
  • enable for everyone else who wants it (talked to cluster-registry, they're now using it) [WIP]
  • enable for kubernetes (rolled out to ci bazel build / test, presubmit test) [WIP]

/area bazel
/area jobs
/assign

[BenTheElder]

Some more notes:

• The invalid cache sharing work around seems to work well with the canary jobs at least
• Experimental Jobs using the cache can be much faster:
  • pull-kubernetes-bazel-test takes 25-30m currently, once the cache is hot pull-kubernetes-bazel-test-canary takes ~5min typically. There is a lot of variation for both though mostly due to the load on the node the job runs on
  • pull-test-infra-bazel takes 8-10m currently, pull-test-infra-bazel-canary takes ~3 min (about two minutes of which is spent installing python deps and running pylint...)
  • pull-kubernetes-bazel-build-canary is not caching well currently, we probably need to mark things like hyperkube and the tarballs as no-cache

[BenTheElder]

FYI @perotinus we can probably look at using this with the cluster-registry soon, test-infra is using it now 😄

[BenTheElder]

Tested eviction a bit more with: https://github.com/BenTheElder/test-infra/blob/20d7d58ac34d59e241eddfb107e3b735398cd8d7/experiment/fill_cache.sh
Will PR some logging changes but so far WAI

Edit: see also, results of turning this on for test-infra:

[BenTheElder]

Metrics dashboard is now up here: http://velodrome.k8s.io/dashboard/db/cache-monitoring?orgId=1

Edit: moved to here http://velodrome.k8s.io/dashboard/db/bazel-cache?orgId=1

[BenTheElder]

Testing a new test-infra PR appears to have 3468 action cache hits, 7 action cache misses, and 1920 CAS hits (!)

[BenTheElder]

I've now turned this on for the kubernetes CI bazel-build and bazel-test jobs with great results*


* Note: the build job only runs in post-submit, and once every 6 hours, currently. Once this job is properly continuous the results for it will be more obvious.

We also have a monitoring dashboard now:

[BenTheElder]

https://k8s-testgrid.appspot.com/presubmits-kubernetes-blocking#pull-kubernetes-bazel-test&graph-metrics=test-duration-minutes

As expected instead of ~25+ minutes we're seeing ~5-6 minutes for pull-kubernetes-bazel-test after switching this on today.

[buchgr]

Absolutely amazing work @BenTheElder. Congratulations!

[BenTheElder]

Thanks Jakob :-)

…

On Fri, Mar 2, 2018 at 8:18 AM Jakob Buchgraber ***@***.***> wrote: Absolutely *amazing* work @BenTheElder <https://github.com/bentheelder>. Congratulations! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6808 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA4Bq3O6fYMjBbZtwdfK8IGwjLzmB7HPks5taXDqgaJpZM4SEbFa> .

[BenTheElder]

kops is now using the remote cache, 8+ minutes to test -> ~2 minutes

https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/pull/kops/4565/pull-kops-bazel-test/

[BenTheElder]

I think once #7205 is in we can close this, I've significantly upped the cache storage and we're flipping it on for pretty much all other presubmits that should leverage caching.

[BenTheElder]

/close
We've rolled this out to many more jobs, pull-kubernetes-bazel-build in particular is now trending towards 5-8 minutes instead of 13+

[buchgr]

woohooo!!! 🍾 🎆 🎉

cc @ulfjack