[Amazon Linux 2] docker buildx create & build error (rootless): mount sysfs:/sys (via /proc/self/fd/6), flags: 0xe: operation not permitted: unknown

[Rez0k]

Hi,

I am running docker in docker rootless (dind-rootless) on kubernetes cluster (eks cluster with containerd as container runtime).
everything is working on dind, but when migrating to dind-rootless I am getting errors while running the commands:

docker buildx create --name ci-builder --config buildkitd.toml --use --driver-opt image=moby/buildkit:v0.10.4-rootless --buildkitd-flags '--security-opt seccomp=unconfined --security-opt apparmor=unconfined --device /dev/fuse --oci-worker-no-process-sandbox --allow-insecure-entitlement security.insecure'

docker buildx build --push --builder ci-builder

[+] Building 0.2s (1/1) FINISHED
=> ERROR [internal] booting buildkit 0.2s
=> => starting container buildx_buildkit_ci-builder0 0.2s

[internal] booting buildkit:

---

ERROR: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "sysfs" to rootfs at "/sys": mount sysfs:/sys (via /proc/self/fd/6), flags: 0xe: operation not permitted: unknown

additional info:
docker info -

Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc., v0.9.1)

Server:
Containers: 2
Running: 0
Paused: 0
Stopped: 2
Images: 2
Server Version: 20.10.18
Storage Driver: fuse-overlayfs
Logging Driver: json-file
Cgroup Driver: none
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
runc version: v1.1.4-0-g5fd4c4d1
init version: de40ad0
Security Options:
seccomp
Profile: default
rootless
Kernel Version: 5.4.209-116.367.amzn2.x86_64
Operating System: Alpine Linux v3.16 (containerized)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.482GiB
Name: jenkins-agent-st3p2
ID: EWAZ:EPLH:6FZS:777F:BT5R:JYGV:PW42:WIBV:7USU:4D6H:FRVC:U4D7
Docker Root Dir: /home/rootless/.local/share/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
docker-registry.docker-registry:5000
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
Access to the remote API is equivalent to root access on the host. Refer
to the 'Docker daemon attack surface' section in the documentation for
more information: https://docs.docker.com/go/attack-surface/
WARNING: Running in rootless-mode without cgroups. To enable cgroups in rootless-mode, you need to boot the system in cgroup v2 mode.

I tried running the dind-rootless with the fuse-overlayfs storage driver and nothing changed.
I also tried modifying the buildkitd-flags and nothing changed.

Thanks !

[AkihiroSuda]

Is this Amazon EKS?

[Rez0k]

Is this Amazon EKS?

@AkihiroSuda yes

[Lavaerius]

I'm also seeing this problem, but on AKS

[lpfann]

We also had the same issue. After debugging a bit what helped for us deactivating Istio namespace injection.

[AkihiroSuda]

Specifying --oci-worker-no-process-sandbox may work (although it seems already specified for the OP's case)

• docs: troubleshoot for `mount proc:/proc (via /proc/self/fd/6), flags… #4075

[AndriyKy]

Hey there! I also have similar issue in IDX when I run docker compose up. Here is the output of the docker version command

Client:
 Version:           24.0.9
 API version:       1.43
 Go version:        go1.21.11
 Git commit:        v24.0.9
 Built:             Thu Jan  1 00:00:00 1970
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          24.0.9
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.21.11
  Git commit:       v24.0.9
  Built:            Tue Jan  1 00:00:00 1980
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.13
  GitCommit:        v1.7.13
 runc:
  Version:          1.1.12
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        
 rootlesskit:
  Version:          1.1.1
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       builtin
  StateDir:         /tmp/rootlesskit3862372010
 slirp4netns:
  Version:          1.2.2
  GitCommit:        0ee2d87523e906518d34a6b423271e4826f71faf

And docker compose version

Docker Compose version 2.23.1