Firecracker: Use docker pull for images #1565
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If
docker_socket
is configured, then when pulling images for firecracker, rundocker pull
followed bydocker save
, rather thanskopeo copy docker://...
. This allows us to reuse locally cached layers from the Docker cache, as well as de-duping across pulls.This should reduce executor warmup time, since on startup we try to pull images for both Docker and Firecracker concurrently.
It should also improve cases where several tasks hit the executor at once with the same container-image that has not yet been pulled, and the tasks would execute extremely slowly (several minutes) due to the same image being downloaded multiple times in parallel, without deduplication. It's unclear how common this will be in production, but I have already run into this several times locally (executor grinding to a halt because it is pulling a bunch of the same image in parallel).
Note that we can't use skopeo to copy directly from the Docker daemon --> OCI image via the
docker-daemon:
source protocol, due to containers/image#1049. As a workaround, we usedocker save
then convert from Docker format to OCI format usingskopeo copy docker-archive:/path/to/docker/save/output.tar oci:/path/to/oci/image:label
This PR is a partial fix for https://github.com/buildbuddy-io/buildbuddy-internal/issues/1097. In future PRs, we would also benefit from de-duping the image conversion (to OCI format), unpacking (umoci), and ext4 image creation as well (mke2fs), since those are all expensive operations (~tens of seconds total).
Version bump: Patch
Related issues: https://github.com/buildbuddy-io/buildbuddy-internal/issues/1097