Pack fails to create ephemeral builder when using Docker with containerd image store

[chenbh]

Summary

When the Docker beta feature for using containerd as the image store is enabled, pack is unable to save ephemeral builders.

---

Reproduction

Steps

1. On Docker Desktop v4.12.0+, enable the containerd image store feature:
   
2. Try to use pack to build an app (i.e. https://github.com/paketo-buildpacks/samples/tree/main/go/mod)

Current behavior

$ pack build --builder paketobuildpacks/builder-jammy-tiny my-app
e1fcf997ef3d: Already exists
5dcbe5c363ba: Already exists
d133c8551fbd: Already exists
3acbe9262199: Already exists
ERROR: failed to build: failed to write image to the following tags: [pack.local/builder/6c776373617a756d7270:latest: saving image "pack.local/builder/6c776373617a756d7270:latest": Error: No such image: 295d34fd324fb6fafe951feb8a8b424a2482e4be75b6859c4136bd22102c36f5]

Expected behavior

It builds the app

---

Environment

pack info

Pack:
  Version:  0.29.0+git-95c8060.build-4209
  OS/Arch:  darwin/arm64

Default Lifecycle Version:  0.15.2

Supported Platform APIs:  0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.10

Config:
  experimental = true
  layout-repo-dir = "/Users/bohanc/.pack/layout-repo"

docker info

Client:
 Version:    24.0.2
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.0
    Path:     /Users/bohanc/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.19.1
    Path:     /Users/bohanc/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/bohanc/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /Users/bohanc/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.6
    Path:     /Users/bohanc/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/bohanc/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/bohanc/.docker/cli-plugins/docker-scan
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  0.16.1
    Path:     /Users/bohanc/.docker/cli-plugins/docker-scout

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 26
 Server Version: 24.0.2-38-g8e70a1b23e
 Storage Driver: stargz
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.49-linuxkit-pr
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 15.61GiB
 Name: docker-desktop
 ID: 1b4957cf-79bc-4ff4-a97b-6c0870ad98ef
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

[chenbh]

I'm not quite sure if this is a bug on pack's or Docker's side, but it would be nice for pack to support this.

The main motivation for not using the Docker image store is that the containerd image store doesn't suffer from the max layer depth exceeded error (well it does, but the limit is something like 500) as seen in #1649 and buildpacks/rfcs#290). .

[jjbustamante]

We want to research the reason for this bug. the idea will be:

• Confirm if there is a bug on pack or if this something related to docker.
• We had some ideas to use the export to OCI layout as a way to interact with containerd (see this issue on lifecycle)
  Once we have done that, we can make some decisions if it worth it to work on this.

Also as a side note, containerd image storage is also experimental in Docker side, I don't want to invest a lot of effort on it until is ready.

[natalieparellano]

Closing in favor of #1519