rootless: mount proc:/proc (via /proc/self/fd/6), flags: 0xe: operation not permitted

[jedevc]

In attempting to try using moby/buildkit:master-rootless I've a small issue with /proc mounting:

$ docker run --security-opt seccomp=unconfined --security-opt apparmor=unconfined --name=buildkitd -p 1234:1234 --rm moby/buildkit:master-rootless --addr tcp://0.0.0.0:1234

The container starts successfully, but on a buildctl --addr tcp://localhost:1234 build --frontend dockerfile.v0 --local dockerfile=. --local context=. of https://github.com/moby/buildkit, the build fails quickly:

[+] Building 2.5s (4/4) FINISHED                                                                                                                           
 => [internal] load build definition from Dockerfile                                                                                                  0.0s
 => => transferring dockerfile: 13.16kB                                                                                                               0.0s
 => resolve image config for docker.io/docker/dockerfile-upstream:master                                                                              1.6s
 => [auth] docker/dockerfile-upstream:pull token for registry-1.docker.io                                                                             0.0s
 => docker-image://docker.io/docker/dockerfile-upstream:master@sha256:acee546d7ab99639a7c924e3e6b7e29d107b3fda0ed53d63ef6f24a452cd55c5                0.8s
 => => resolve docker.io/docker/dockerfile-upstream:master@sha256:acee546d7ab99639a7c924e3e6b7e29d107b3fda0ed53d63ef6f24a452cd55c5                    0.0s
 => => sha256:4ab15ed2917705e9b80323cd60396d4ba7abb500ea599b41d62b4e40ac06937c 11.80MB / 11.80MB                                                      0.7s
 => => extracting sha256:4ab15ed2917705e9b80323cd60396d4ba7abb500ea599b41d62b4e40ac06937c                                                             0.1s
Dockerfile:1
--------------------
   1 | >>> # syntax=docker/dockerfile-upstream:master
   2 |     
   3 |     ARG RUNC_VERSION=v1.1.7
--------------------
error: failed to solve: exit code: 1

In the buildkit logs:

time="2023-07-28T09:13:15Z" level=info msg="auto snapshotter: using overlayfs"
time="2023-07-28T09:13:15Z" level=info msg="found worker \"foyp8ifilfv1o45yc7yse77or\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:8adbf8b61121 org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.oci.process-mode:sandbox org.mobyproject.buildkit.worker.selinux.enabled:false org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/amd64/v2 linux/amd64/v3 linux/386]"
time="2023-07-28T09:13:15Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
time="2023-07-28T09:13:15Z" level=info msg="found 1 workers, default=\"foyp8ifilfv1o45yc7yse77or\""
time="2023-07-28T09:13:15Z" level=warning msg="currently, only the default worker can be used."
time="2023-07-28T09:13:15Z" level=warning msg="TLS is not enabled for tcp://0.0.0.0:1235. enabling mutual TLS authentication is highly recommended"
time="2023-07-28T09:13:15Z" level=info msg="running server on [::]:1235"
runc run failed: unable to start container process: error during container init: error mounting "proc" to rootfs at "/proc": mount proc:/proc (via /proc/self/fd/6), flags: 0xe: operation not permitted
time="2023-07-28T09:13:19Z" level=error msg="/moby.buildkit.v1.frontend.LLBBridge/Solve returned error: rpc error: code = Unknown desc = exit code: 1"
time="2023-07-28T09:13:19Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = exit code: 1"

I did some basic investigation, it seems slightly reminiscent of opencontainers/runc#1658? Based on that, changing the docker run to include -v /proc:/newproc does in-fact "fix" the issue.

Maybe also related to moby/moby#41893? Applying a similar patch in https://github.com/moby/buildkit also "fixes" the issue (though I'm actually unsure if this is the fix wanted):

diff --git a/util/rootless/specconv/specconv_linux.go b/util/rootless/specconv/specconv_linux.go
index 7118f8d6d..5c5beca84 100644
--- a/util/rootless/specconv/specconv_linux.go
+++ b/util/rootless/specconv/specconv_linux.go
@@ -1,6 +1,7 @@
 package specconv
 
 import (
+	"path"
 	"strings"
 
 	specs "github.com/opencontainers/runtime-spec/specs-go"
@@ -29,10 +30,27 @@ func ToRootless(spec *specs.Spec) error {
 		if strings.HasPrefix(mount.Destination, "/sys") {
 			continue
 		}
+		if path.Clean(mount.Destination) == "/proc" {
+			mount = specs.Mount{
+				Destination: "/proc",
+				Type:        "bind",
+				Source:      "/proc",
+				Options:     []string{"rbind", "nosuid", "noexec", "nodev"},
+			}
+		}
 		mounts = append(mounts, mount)
 	}
 	spec.Mounts = mounts
 
+	// Remove ReadonlyPaths for /proc/*
+	newROP := spec.Linux.ReadonlyPaths[:0]
+	for _, x := range spec.Linux.ReadonlyPaths {
+		x = path.Clean(x)
+		if !strings.HasPrefix(x, "/proc/") {
+			newROP = append(newROP, x)
+		}
+	}
+
 	// Remove cgroups so as to avoid `container_linux.go:337: starting container process caused "process_linux.go:280: applying cgroup configuration for process caused \"mkdir /sys/fs/cgroup/cpuset/buildkit: permission denied\""`
 	spec.Linux.Resources = nil
 	spec.Linux.CgroupsPath = ""

System info

$ uname -r
6.4.3-arch1-1
$ pacman -Si linux
Repository      : core
Name            : linux
Version         : 6.4.3.arch1-1
Description     : The Linux kernel and modules
Architecture    : x86_64
URL             : https://github.com/archlinux/linux/commits/v6.4.3-arch1
Licenses        : GPL2
Groups          : None
Provides        : KSMBD-MODULE  VIRTUALBOX-GUEST-MODULES  WIREGUARD-MODULE
Depends On      : coreutils  initramfs  kmod
Optional Deps   : wireless-regdb: to set the correct wireless channels of your country
                  linux-firmware: firmware images needed for some devices
Conflicts With  : None
Replaces        : virtualbox-guest-modules-arch  wireguard-arch
Download Size   : 125.41 MiB
Installed Size  : 125.47 MiB
Packager        : Jan Alexander Steffens (heftig) <[email protected]>
Build Date      : Tue 11 Jul 2023 06:13:39 AM BST
Validated By    : MD5 Sum  SHA-256 Sum  Signature

$ docker info
Client:
 Version:    24.0.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.11.0
    Path:     /usr/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.20.0
    Path:     /usr/lib/docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /usr/lib/docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /usr/lib/docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.6
    Path:     /usr/lib/docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /usr/lib/docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /usr/lib/docker/cli-plugins/docker-scan
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  0.16.1
    Path:     /usr/lib/docker/cli-plugins/docker-scout

Server:
 Containers: 2
  Running: 0
  Paused: 0
  Stopped: 2
 Images: 17
 Server Version: 24.0.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 0cae528dd6cb557f7201036e9f43420650207b58.m
 runc version: 
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.4.3-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 30.72GiB
 Name: docker-dragon
 ID: JBHZ:KCJU:UUEX:PXZU:ELI2:FHKJ:MEIC:TKSP:E7VN:FD6N:2AIN:PQIS
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  localhost:443
  127.0.0.0/8
 Live Restore Enabled: false

Happy to try and make a VM image to reproduce my system if necessary 😄

[AkihiroSuda]

-v /var/lib/buildkit:/var/lib/buildkit isn’t correct setup, especially for rootless

[jedevc]

Ah correct, yes, my bad - issue still reproduces without that, so I've removed it from the examples above.

[AkihiroSuda]

You need to pass --oci-worker-no-process-sandbox to buildkitd

https://github.com/moby/buildkit/blob/v0.12/docs/rootless.md#docker

[jedevc]

🤦 🤦 🤦 welp. That's it. Apparently, I can't read.

Thanks @AkihiroSuda, sorry to bother you.

[AkihiroSuda]

• docs: troubleshoot for `mount proc:/proc (via /proc/self/fd/6), flags… #4075