Cache CI Docker images in ghcr registry #119290

Kobzol · 2023-12-25T10:14:10Z

This PR changes the way rust-lang caches Docker images used in CI workflows. Before, the intermediate Docker layers were manually exported from docker history and backed up in S3. However, this approach doesn't work any more with the Docker version used by GitHub Actions since August 2023. We had to revert to disabling Docker BuildKit to make the old caching work, but this workaround will stop working eventually, after GitHub updates Docker again and the old build backend will be removed.

This PR changes the caching to use Docker caching instead. There are several backends for the cache, for our use-case S3 and Docker registry makes sense. This PR uses the Docker registry backend and uses the ghcr.io registry.

The caching creates a Docker image labeled rust-ci, which is currently stored to the ghcr.io/rust-lang-ci package registry. This image appears here. The image is stored in rust-lang-ci and not rust-lang, because try and auto builds run in the context of that repository, so the used GITHUB_TOKEN has permissions for it (unlike for rust-lang).

For pull request CI runs, the provided GITHUB_TOKEN reduces its permissions automatically to packages: read, which means that we won't be able to write the Docker image. If we're not able to write, we won't have anything to read. So I disabled the caching entirely for PR runs (it makes it slightly faster to build the Docker image if we don't have to deal with exporting and using a separate build driver). Note that before this PR, we also weren't able to read or write the cache on PR runs.

Rustup part of this change is here.

Related issue: rust-lang/infra-team#81

r? @Mark-Simulacrum

Mark-Simulacrum · 2023-12-29T14:54:40Z

@bors try

This broadly looks fine to me -- is there a reason I don't see try builds etc. on this PR? I think we should test out that it works on try + local builds (src/ci/docker/run.sh from your laptop or w/e), and perhaps get rustup CI working, and then we can merge.

bors · 2023-12-29T14:55:50Z

⌛ Trying commit 6f90144 with merge a4cbdba...

…try> Cache CI Docker images in ghcr registry This PR changes the way `rust-lang` caches Docker images used in CI workflows. Before, the intermediate Docker layers were manually exported from `docker history` and backed up in S3. However, this approach doesn't work any more with the Docker version used by GitHub Actions since August 2023. We had to revert to disabling Docker BuildKit to make the old caching work, but this workaround will stop working eventually, after GitHub updates Docker again and the old build backend will be removed. This PR changes the caching to use [Docker caching](https://docs.docker.com/build/cache/) instead. There are several backends for the cache, for our use-case S3 and Docker registry makes sense. This PR uses the Docker registry backend and uses the ghcr.io registry. The caching creates a Docker image labeled `rust-ci`, which is currently stored to the `ghcr.io/rust-lang-ci` package registry. This image appears [here](https://ghcr.io/rust-lang-ci/rust-ci). The image is stored in `rust-lang-ci` and not `rust-lang`, because `try` and `auto` builds run in the context of that repository, so the used `GITHUB_TOKEN` has permissions for it (unlike for `rust-lang`). For pull request CI runs, the provided `GITHUB_TOKEN` reduces its permissions automatically to `packages: read`, which means that we won't be able to write the Docker image. If we're not able to write, we won't have anything to read. So I disabled the caching entirely for PR runs (it makes it slightly faster to build the Docker image if we don't have to deal with exporting and using a separate build driver). Note that before this PR, we also weren't able to read or write the cache on PR runs. Related issue: rust-lang/infra-team#81 r? `@Mark-Simulacrum`

Kobzol · 2023-12-29T14:57:58Z

is there a reason I don't see try builds etc. on this PR

I tested a try build on a different PR, but we should of course also try it here.

What I'm struggling with currently is how to use the cache as a docker pull-able image from rustup. It doesn't seem to be possible to pull the cached images from the repository.

bors · 2023-12-29T16:24:40Z

☀️ Try build successful - checks-actions
Build commit: a4cbdba (a4cbdbac0b250cb863eca2c0f95c92800fc3ed6c)

Kobzol · 2023-12-29T16:55:59Z

@bors try

bors · 2023-12-29T16:57:09Z

⌛ Trying commit 6f90144 with merge 180eb9e...

…try> Cache CI Docker images in ghcr registry This PR changes the way `rust-lang` caches Docker images used in CI workflows. Before, the intermediate Docker layers were manually exported from `docker history` and backed up in S3. However, this approach doesn't work any more with the Docker version used by GitHub Actions since August 2023. We had to revert to disabling Docker BuildKit to make the old caching work, but this workaround will stop working eventually, after GitHub updates Docker again and the old build backend will be removed. This PR changes the caching to use [Docker caching](https://docs.docker.com/build/cache/) instead. There are several backends for the cache, for our use-case S3 and Docker registry makes sense. This PR uses the Docker registry backend and uses the ghcr.io registry. The caching creates a Docker image labeled `rust-ci`, which is currently stored to the `ghcr.io/rust-lang-ci` package registry. This image appears [here](https://ghcr.io/rust-lang-ci/rust-ci). The image is stored in `rust-lang-ci` and not `rust-lang`, because `try` and `auto` builds run in the context of that repository, so the used `GITHUB_TOKEN` has permissions for it (unlike for `rust-lang`). For pull request CI runs, the provided `GITHUB_TOKEN` reduces its permissions automatically to `packages: read`, which means that we won't be able to write the Docker image. If we're not able to write, we won't have anything to read. So I disabled the caching entirely for PR runs (it makes it slightly faster to build the Docker image if we don't have to deal with exporting and using a separate build driver). Note that before this PR, we also weren't able to read or write the cache on PR runs. Related issue: rust-lang/infra-team#81 r? `@Mark-Simulacrum`

bors · 2023-12-29T18:23:08Z

☀️ Try build successful - checks-actions
Build commit: 180eb9e (180eb9e1b26bc11a3ca55fbebfda4b32af2541f4)

Mark-Simulacrum · 2023-12-31T23:02:34Z

@rustbot author

Let me know if I can/should try to help here, otherwise will put this in waiting on author for now.

Kobzol · 2024-01-21T13:31:38Z

@bors try

bors · 2024-01-21T13:32:48Z

⌛ Testing commit 8bb6a1b with merge 0872387...

Cache CI Docker images in ghcr registry This PR changes the way `rust-lang` caches Docker images used in CI workflows. Before, the intermediate Docker layers were manually exported from `docker history` and backed up in S3. However, this approach doesn't work any more with the Docker version used by GitHub Actions since August 2023. We had to revert to disabling Docker BuildKit to make the old caching work, but this workaround will stop working eventually, after GitHub updates Docker again and the old build backend will be removed. This PR changes the caching to use [Docker caching](https://docs.docker.com/build/cache/) instead. There are several backends for the cache, for our use-case S3 and Docker registry makes sense. This PR uses the Docker registry backend and uses the ghcr.io registry. The caching creates a Docker image labeled `rust-ci`, which is currently stored to the `ghcr.io/rust-lang-ci` package registry. This image appears [here](https://ghcr.io/rust-lang-ci/rust-ci). The image is stored in `rust-lang-ci` and not `rust-lang`, because `try` and `auto` builds run in the context of that repository, so the used `GITHUB_TOKEN` has permissions for it (unlike for `rust-lang`). For pull request CI runs, the provided `GITHUB_TOKEN` reduces its permissions automatically to `packages: read`, which means that we won't be able to write the Docker image. If we're not able to write, we won't have anything to read. So I disabled the caching entirely for PR runs (it makes it slightly faster to build the Docker image if we don't have to deal with exporting and using a separate build driver). Note that before this PR, we also weren't able to read or write the cache on PR runs. Related issue: rust-lang/infra-team#81 r? `@Mark-Simulacrum`

Kobzol · 2024-01-21T13:41:54Z

@bors try

Kobzol · 2024-01-21T13:42:11Z

The Rustup part of this change is at: rust-lang/rustup#3648. Local build seems to work.

bors · 2024-01-27T22:23:31Z

⌛ Testing commit da5ab5e with merge ea585a9...

…ark-Simulacrum Cache CI Docker images in ghcr registry This PR changes the way `rust-lang` caches Docker images used in CI workflows. Before, the intermediate Docker layers were manually exported from `docker history` and backed up in S3. However, this approach doesn't work any more with the Docker version used by GitHub Actions since August 2023. We had to revert to disabling Docker BuildKit to make the old caching work, but this workaround will stop working eventually, after GitHub updates Docker again and the old build backend will be removed. This PR changes the caching to use [Docker caching](https://docs.docker.com/build/cache/) instead. There are several backends for the cache, for our use-case S3 and Docker registry makes sense. This PR uses the Docker registry backend and uses the ghcr.io registry. The caching creates a Docker image labeled `rust-ci`, which is currently stored to the `ghcr.io/rust-lang-ci` package registry. This image appears [here](https://ghcr.io/rust-lang-ci/rust-ci). The image is stored in `rust-lang-ci` and not `rust-lang`, because `try` and `auto` builds run in the context of that repository, so the used `GITHUB_TOKEN` has permissions for it (unlike for `rust-lang`). For pull request CI runs, the provided `GITHUB_TOKEN` reduces its permissions automatically to `packages: read`, which means that we won't be able to write the Docker image. If we're not able to write, we won't have anything to read. So I disabled the caching entirely for PR runs (it makes it slightly faster to build the Docker image if we don't have to deal with exporting and using a separate build driver). Note that before this PR, we also weren't able to read or write the cache on PR runs. Rustup part of this change is [here](rust-lang/rustup#3648). Related issue: rust-lang/infra-team#81 r? `@Mark-Simulacrum`

bors · 2024-01-27T22:25:30Z

💔 Test failed - checks-actions

Kobzol · 2024-01-28T16:06:08Z

@bors r=Mark-simulacrum

(This run will fail, but I added additional logging to help me debug the used Docker version on our self-hosted runners).

bors · 2024-01-28T16:06:10Z

📌 Commit 09e0d4f has been approved by Mark-simulacrum

It is now in the queue for this repository.

bors · 2024-01-29T02:51:00Z

⌛ Testing commit 09e0d4f with merge 7e43442...

bors · 2024-01-29T05:13:45Z

☀️ Test successful - checks-actions
Approved by: Mark-simulacrum
Pushing 7e43442 to master...

rust-timer · 2024-01-29T06:28:44Z

Finished benchmarking commit (7e43442): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.9%	[2.9%, 2.9%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-3.2%	[-3.4%, -3.0%]	2
All ❌✅ (primary)	2.9%	[2.9%, 2.9%]	1

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 660.85s -> 661.466s (0.09%)
Artifact size: 308.02 MiB -> 308.03 MiB (0.00%)

…acrum CI: cache PR CI Docker builds An attempt to restore Docker caching for PR CI workflows that share the Docker image with an auto/try workflow. This was broken by my previous [PR](rust-lang#119290) that changed how we cache the Docker builds. [Before](rust-lang#122545): ![image](https://github.com/rust-lang/rust/assets/4539057/05e0d347-af64-4e85-bc99-0e4ac07192ec) After: ![image](https://github.com/rust-lang/rust/assets/4539057/2f657d60-b242-45eb-ac61-d8f71787edda) r? `@ghost`

…acrum CI: cache PR CI Docker builds An attempt to restore Docker caching for PR CI workflows that share the Docker image with an auto/try workflow. This was broken by my previous [PR](rust-lang#119290) that changed how we cache the Docker builds. [Before](rust-lang#122545): ![image](https://github.com/rust-lang/rust/assets/4539057/05e0d347-af64-4e85-bc99-0e4ac07192ec) After: ![image](https://github.com/rust-lang/rust/assets/4539057/2f657d60-b242-45eb-ac61-d8f71787edda) r? ``@ghost``

Rollup merge of rust-lang#122563 - Kobzol:ci-pr-caching, r=Mark-Simulacrum CI: cache PR CI Docker builds An attempt to restore Docker caching for PR CI workflows that share the Docker image with an auto/try workflow. This was broken by my previous [PR](rust-lang#119290) that changed how we cache the Docker builds. [Before](rust-lang#122545): ![image](https://github.com/rust-lang/rust/assets/4539057/05e0d347-af64-4e85-bc99-0e4ac07192ec) After: ![image](https://github.com/rust-lang/rust/assets/4539057/2f657d60-b242-45eb-ac61-d8f71787edda) r? ``@ghost``

rustbot assigned Mark-Simulacrum Dec 25, 2023

rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Dec 25, 2023

This comment has been minimized.

Sign in to view

Kobzol force-pushed the ci-docker-registry-cache branch from 9e9604b to 6f90144 Compare December 25, 2023 10:29

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 31, 2023

Kobzol force-pushed the ci-docker-registry-cache branch 2 times, most recently from 7dae399 to 41ef4a5 Compare January 21, 2024 13:29

Kobzol force-pushed the ci-docker-registry-cache branch from 41ef4a5 to 8bb6a1b Compare January 21, 2024 13:32

Kobzol mentioned this pull request Jan 21, 2024

[WIP] CI: use BuildKit Docker layer caching #114762

Closed

Kobzol force-pushed the ci-docker-registry-cache branch from 8bb6a1b to 53a9eae Compare January 21, 2024 13:39

Kobzol mentioned this pull request Jan 21, 2024

Download rust CI Docker images from a registry rust-lang/rustup#3648

Merged

This comment has been minimized.

Sign in to view

bors added the S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. label Jan 27, 2024

This comment has been minimized.

Sign in to view

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jan 27, 2024

Mark-Simulacrum added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 28, 2024

Print image input file and Docker version

09e0d4f

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jan 28, 2024

Kobzol mentioned this pull request Jan 28, 2024

Add support for Docker BuildKit rust-lang/gha-self-hosted#19

Merged

bors added the merged-by-bors This PR was explicitly merged by bors. label Jan 29, 2024

bors merged commit 7e43442 into rust-lang:master Jan 29, 2024
12 checks passed

rustbot added this to the 1.77.0 milestone Jan 29, 2024

Kobzol deleted the ci-docker-registry-cache branch January 29, 2024 07:40

This was referenced Feb 13, 2024

Implement Docker caching on top of BuildKit rust-lang/infra-team#81

Closed

Optimize Docker caching on CI rust-lang/infra-team#103

Open

Kobzol mentioned this pull request Mar 15, 2024

CI: cache PR CI Docker builds #122563

Merged

Kobzol mentioned this pull request May 6, 2024

Rewrite CI documentation rust-lang/rustc-dev-guide#1972

Merged

ehuss mentioned this pull request Oct 26, 2024

Create a GitHub token for publishing (Docker) packages rust-lang/infra-team#95

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cache CI Docker images in ghcr registry #119290

Cache CI Docker images in ghcr registry #119290

Kobzol commented Dec 25, 2023 •

edited

Loading

This comment has been minimized.

Mark-Simulacrum commented Dec 29, 2023

bors commented Dec 29, 2023

Kobzol commented Dec 29, 2023

bors commented Dec 29, 2023

Kobzol commented Dec 29, 2023

bors commented Dec 29, 2023

bors commented Dec 29, 2023

Mark-Simulacrum commented Dec 31, 2023

Kobzol commented Jan 21, 2024

bors commented Jan 21, 2024

Kobzol commented Jan 21, 2024

Kobzol commented Jan 21, 2024

This comment has been minimized.

bors commented Jan 27, 2024

This comment has been minimized.

bors commented Jan 27, 2024

Kobzol commented Jan 28, 2024

bors commented Jan 28, 2024

bors commented Jan 29, 2024

bors commented Jan 29, 2024

rust-timer commented Jan 29, 2024

Cache CI Docker images in ghcr registry #119290

Cache CI Docker images in ghcr registry #119290

Conversation

Kobzol commented Dec 25, 2023 • edited Loading

This comment has been minimized.

Mark-Simulacrum commented Dec 29, 2023

bors commented Dec 29, 2023

Kobzol commented Dec 29, 2023

bors commented Dec 29, 2023

Kobzol commented Dec 29, 2023

bors commented Dec 29, 2023

bors commented Dec 29, 2023

Mark-Simulacrum commented Dec 31, 2023

Kobzol commented Jan 21, 2024

bors commented Jan 21, 2024

Kobzol commented Jan 21, 2024

Kobzol commented Jan 21, 2024

This comment has been minimized.

bors commented Jan 27, 2024

This comment has been minimized.

bors commented Jan 27, 2024

Kobzol commented Jan 28, 2024

bors commented Jan 28, 2024

bors commented Jan 29, 2024

bors commented Jan 29, 2024

rust-timer commented Jan 29, 2024

Overall result: no relevant changes - no action needed

Instruction count

Max RSS (memory usage)

Cycles

Binary size

Kobzol commented Dec 25, 2023 •

edited

Loading