Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Firecracker: Use docker pull for images #1565

Merged
merged 11 commits into from
Feb 14, 2022
Merged

Conversation

bduffany
Copy link
Member

@bduffany bduffany commented Feb 10, 2022

If docker_socket is configured, then when pulling images for firecracker, run docker pull followed by docker save, rather than skopeo copy docker://.... This allows us to reuse locally cached layers from the Docker cache, as well as de-duping across pulls.

This should reduce executor warmup time, since on startup we try to pull images for both Docker and Firecracker concurrently.

It should also improve cases where several tasks hit the executor at once with the same container-image that has not yet been pulled, and the tasks would execute extremely slowly (several minutes) due to the same image being downloaded multiple times in parallel, without deduplication. It's unclear how common this will be in production, but I have already run into this several times locally (executor grinding to a halt because it is pulling a bunch of the same image in parallel).

Note that we can't use skopeo to copy directly from the Docker daemon --> OCI image via the docker-daemon: source protocol, due to containers/image#1049. As a workaround, we use docker save then convert from Docker format to OCI format using skopeo copy docker-archive:/path/to/docker/save/output.tar oci:/path/to/oci/image:label

This PR is a partial fix for https://github.com/buildbuddy-io/buildbuddy-internal/issues/1097. In future PRs, we would also benefit from de-duping the image conversion (to OCI format), unpacking (umoci), and ext4 image creation as well (mke2fs), since those are all expensive operations (~tens of seconds total).


Version bump: Patch

Related issues: https://github.com/buildbuddy-io/buildbuddy-internal/issues/1097

@bduffany bduffany changed the title Use docker pull for firecracker images Firecracker: Use docker pull for images Feb 10, 2022
Copy link
Member

@vadimberezniker vadimberezniker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really nice.

@bduffany bduffany merged commit c0952f1 into master Feb 14, 2022
@bduffany bduffany deleted the firecracker-docker-pull branch February 14, 2022 17:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants