Buildx not work properly on Mac with Apple Silicon chip #6928

amizurov · 2023-06-18T12:13:16Z

Contributing guidelines

I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

... the documentation does not mention anything about my problem
... there are no open or closed issues that are related to my problem

Description

We use https://github.com/fabric8io/fabric8-maven-plugin to create multi-platform architecture images. And everything works correctly on Docker Desktop with Mac and Intel chip (amd64), but when we run build on Mac with Silicon chip (arm64) we got the error (125):

 docker --config /Users/myuser/projects/container_template/base-image-ubuntu/target/docker-output/docker.artifactory.mycompany.com/containers/base-image-ubuntu/1.1.0/docker buildx create --driver docker-container --name maven
[INFO] DOCKER> unknown flag: --driver
[INFO] DOCKER> See 'docker --help'.
[INFO] DOCKER> 
[INFO] DOCKER> Usage:  docker [OPTIONS] COMMAND
[INFO] DOCKER> 
[INFO] DOCKER> A self-sufficient runtime for containers
...
 DOCKER> Error status (125) while creating builder maven

Also i execute the simple command (docker --config /tmp/docker buildx) and receive difference results.
Apple Silicon:

docker --config /tmp/docker buildx ls
docker: 'buildx' is not a docker command.
See 'docker --help'

Apple Intel:

Usage:  docker buildx [OPTIONS] COMMAND

Extended build capabilities with BuildKit

Options:
      --builder string   Override the configured builder instance

Management Commands:
  imagetools  Commands to work on images in registry

Commands:
  bake        Build from a file
  build       Start a build
  create      Create a new builder instance
  du          Disk usage
  inspect     Inspect current builder instance
  ls          List builder instances
  prune       Remove build cache
  rm          Remove a builder instance
  stop        Stop builder instance
  use         Set the current builder instance
  version     Show buildx version information

Run 'docker buildx COMMAND --help' for more information on a command.

Both devices have the same version of Docker Desktop.

Expected behaviour

No exceptions when using buildx, the behavior is the same for both architectures.

Actual behaviour

See description

Buildx version

github.com/docker/buildx v0.10.5 86bdced7766639d56baa4c7c449a4f6468490f87

Docker info

Client:
 Version:    24.0.2
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.5
    Path:     /Users/myuser/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.18.1
    Path:     /Users/myuser/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/myuser/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.19
    Path:     /Users/myuser/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.4
    Path:     /Users/myuser/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/myuser/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/myuser/.docker/cli-plugins/docker-scan
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  v0.12.0
    Path:     /Users/myuser/.docker/cli-plugins/docker-scout

Server:
 Containers: 5
  Running: 0
  Paused: 0
  Stopped: 5
 Images: 716
 Server Version: 24.0.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.49-linuxkit-pr
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 7.765GiB
 Name: docker-desktop
 ID: 305ec8ab-f713-4696-8916-0a1dabb007da
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

Builders list

NAME/NODE       DRIVER/ENDPOINT             STATUS   BUILDKIT                              PLATFORMS
maven           docker-container
  maven0        unix:///var/run/docker.sock inactive
default         docker
  default       default                     running  v0.11.7-0.20230525183624-798ad6b0ce9f linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
desktop-linux * docker
  desktop-linux desktop-linux               running  v0.11.7-0.20230525183624-798ad6b0ce9f linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

Configuration


docker --config /tmp/docker buildx

Build logs

No response

Additional info

No response

The text was updated successfully, but these errors were encountered:

crazy-max · 2023-07-17T09:14:37Z

What's the outcome of the following command?:

$ docker buildx create --driver docker-container --name maven

docker --config /Users/myuser/projects/container_template/base-image-ubuntu/target/docker-output/docker.artifactory.mycompany.com/containers/base-image-ubuntu/1.1.0/docker buildx create --driver docker-container --name maven
[INFO] DOCKER> unknown flag: --driver
[INFO] DOCKER> See 'docker --help'.
[INFO] DOCKER>
[INFO] DOCKER> Usage: docker [OPTIONS] COMMAND
[INFO] DOCKER>
[INFO] DOCKER> A self-sufficient runtime for containers
...
DOCKER> Error status (125) while creating builder maven

Looks like this config folder is broken. What's the output of:

docker --config /Users/myuser/projects/container_template/base-image-ubuntu/target/docker-output/docker.artifactory.mycompany.com/containers/base-image-ubuntu/1.1.0/docker info

Also i execute the simple command (docker --config /tmp/docker buildx) and receive difference results. Apple Silicon:
docker --config /tmp/docker buildx ls
docker: 'buildx' is not a docker command.
See 'docker --help'

I'm not able to repro as well:

docker --config /tmp/docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS  BUILDKIT                              PLATFORMS
default * docker
  default default         running v0.11.7-0.20230525183624-798ad6b0ce9f linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

What's the output of:

docker --config /tmp/docker info

Closing this one as it seems an issue with your environment.

amizurov · 2023-07-27T09:54:42Z

@crazy-max Hey, Thanks for the reaction of this issue, but i think you cannot reproduce it because you run docker --config /tmp/docker buildx ls on amd64 architecture. I have reproduces this on at least 4 machines (Apple M1 Pro MacOS Ventura 13.5). If you think that the problem with environments i will be really appreciate any help to identify it
and please reopen this issue because it not closed, thanks.

JosephZelada · 2023-07-27T21:38:05Z

I'll echo that I'm experiencing this same issue on an M2 Max Mac on macOS Ventura 13.5. A fix or guidance would be deeply appreciated, as I'm trying to get one of our legacy applications running with this, specifically with the docker-maven-plugin

crazy-max · 2023-07-27T23:11:25Z

@amizurov No issue for me on M1:

$ docker --config /tmp/docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS  BUILDKIT             PLATFORMS
default * docker                                       
  default default         running v0.11.6+0a15675913b7 linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6

$ uname -a
Darwin mac-mini-m1.home.foo.com 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun  8 22:22:19 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T8103 arm64

What version of Docker Desktop btw?

nicks · 2023-07-28T00:17:58Z

I believe --config defaults to ~/.docker, so that buildx is loaded from ~/.docker/cli-plugins.

If you set --config=/tmp/docker, it will stop looking in ~/.docker/cli-plugins, and fall back to looking in the system directories for buildx.

From your docker info, I'm guessing you only have buildx installed in ~/.docker/cli-plugins, so setting this flag breaks your env.

JosephZelada · 2023-07-28T01:35:25Z

Mine is 4.20.1

More output for the curious

$ docker --config /tmp/docker buildx ls
docker: 'buildx' is not a docker command.
See 'docker --help'
$ uname -a
Darwin ML-???? 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun  8 22:22:23 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T6020 arm64
$

amizurov · 2023-07-28T08:48:31Z

What version of Docker Desktop btw?

Version 4.21.1 (114176)

uname -a
Darwin macBook 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul  5 22:22:05 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T6000 arm64

amizurov · 2023-07-28T08:58:16Z

I believe --config defaults to ~/.docker, so that buildx is loaded from ~/.docker/cli-plugins.

If you set --config=/tmp/docker, it will stop looking in ~/.docker/cli-plugins, and fall back to looking in the system directories for buildx.

From your docker info, I'm guessing you only have buildx installed in ~/.docker/cli-plugins, so setting this flag breaks your env.

I have the same setup on my MacBook amd64 and docker --config /tmp/docker buildx ls works properly on it. I'm agreed that something wrong with --config command but i have no idea what exactly and only on aarch64.

crazy-max · 2023-07-28T12:49:23Z

@amizurov @JosephZelada

What's the output of ls -al /usr/local/lib/docker/cli-plugins/ ?

FWIW we are checking plugins in multiple directories: https://github.com/docker/cli/blob/e0e27724390cbce21cf6d67568972a4227b07382/cli-plugins/manager/manager_unix.go#L5-L8

In my case they are located in both /usr/local/lib/docker/cli-plugins/ and /Users/<name>/.docker/cli-plugins.

amizurov · 2023-07-28T12:57:02Z

@crazy-max On my amd64 machine ls -al /usr/local/lib/docker/cli-plugins/:

docker-buildx
docker-compose
...

But on aarch64 ls -al /usr/local/lib/docker/cli-plugins/:

No such file or directory

JosephZelada · 2023-07-28T16:26:12Z

Same "No such file or directory" result running that ls -al on my machine

crazy-max · 2023-07-28T16:29:51Z

Thanks for your feedback. Reopening and moving this issue to docker/for-mac repo as this is linked to plugin handling on Docker Desktop.

JosephZelada · 2023-08-02T14:11:05Z

@crazy-max This should have the status/triage label put back on it, shouldn't it?

drcca · 2023-09-20T21:20:25Z

In case this helps anyone. I ran into this issue with these versions:

Docker version 24.0.5, build ced0996
Docker desktop 4.22.1 (118664)

I then upgraded to the latest and greatest 4.23.0 and 24.0.6 and that still didn't help.

A colleague had a setup that worked but was on an older version. One major difference was that he had this symlink (as alluded to above).
/usr/local/lib/docker/cli-plugins -> /Applications/Docker.app/Contents/Resources/cli-plugins
Once I created that symlink stuff started to work again. Not sure if this is correct or could end up masking other problems but I decided to put it out there... (I'm sure most people have cli-plugins living under ~/.docker/cli-plugins which symlink to the same directory under /Applications)

zburgermeiszter · 2024-02-02T16:59:24Z

@drcca you saved my day!
Thanks.

amizurov added the status/triage label Jun 18, 2023

amizurov mentioned this issue Jun 18, 2023

Make --config from buildx command string generation optional fabric8io/docker-maven-plugin#1673

Merged

crazy-max closed this as completed Jul 17, 2023

crazy-max removed the status/triage label Jul 17, 2023

crazy-max reopened this Jul 28, 2023

crazy-max transferred this issue from docker/buildx Jul 28, 2023

crazy-max added the needs-triage label Aug 2, 2023

pdurbin mentioned this issue Aug 10, 2023

DOCKER> unknown flag: --driver with M1 Mac IQSS/dataverse#9771

Closed

riisi mentioned this issue Jan 5, 2024

Issues with buildx pantsbuild/pants#20189

Closed

rohanKanojia mentioned this issue Feb 3, 2024

Multi-Architecture Builds Succeed But Fail to Push fabric8io/docker-maven-plugin#1709

Closed

bsousaa added area/cli status/triage and removed needs-triage labels Feb 5, 2024

rohanKanojia mentioned this issue Feb 11, 2024

fix (buildx) : BuildX failing for Docker CLI on MacOS fabric8io/docker-maven-plugin#1754

Merged

nijave mentioned this issue Mar 26, 2024

Buildx unknown flag: --output when HOME env var is missing #7228

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Buildx not work properly on Mac with Apple Silicon chip #6928

Buildx not work properly on Mac with Apple Silicon chip #6928

amizurov commented Jun 18, 2023 •

edited

Loading

crazy-max commented Jul 17, 2023

amizurov commented Jul 27, 2023 •

edited

Loading

JosephZelada commented Jul 27, 2023

crazy-max commented Jul 27, 2023

nicks commented Jul 28, 2023 •

edited

Loading

JosephZelada commented Jul 28, 2023 •

edited

Loading

amizurov commented Jul 28, 2023

amizurov commented Jul 28, 2023

crazy-max commented Jul 28, 2023

amizurov commented Jul 28, 2023

JosephZelada commented Jul 28, 2023

crazy-max commented Jul 28, 2023

JosephZelada commented Aug 2, 2023

drcca commented Sep 20, 2023 •

edited

Loading

zburgermeiszter commented Feb 2, 2024

Buildx not work properly on Mac with Apple Silicon chip #6928

Buildx not work properly on Mac with Apple Silicon chip #6928

Comments

amizurov commented Jun 18, 2023 • edited Loading

Contributing guidelines

I've found a bug and checked that ...

Description

Expected behaviour

Actual behaviour

Buildx version

Docker info

Builders list

Configuration

Build logs

Additional info

crazy-max commented Jul 17, 2023

amizurov commented Jul 27, 2023 • edited Loading

JosephZelada commented Jul 27, 2023

crazy-max commented Jul 27, 2023

nicks commented Jul 28, 2023 • edited Loading

JosephZelada commented Jul 28, 2023 • edited Loading

amizurov commented Jul 28, 2023

amizurov commented Jul 28, 2023

crazy-max commented Jul 28, 2023

amizurov commented Jul 28, 2023

JosephZelada commented Jul 28, 2023

crazy-max commented Jul 28, 2023

JosephZelada commented Aug 2, 2023

drcca commented Sep 20, 2023 • edited Loading

zburgermeiszter commented Feb 2, 2024

amizurov commented Jun 18, 2023 •

edited

Loading

amizurov commented Jul 27, 2023 •

edited

Loading

nicks commented Jul 28, 2023 •

edited

Loading

JosephZelada commented Jul 28, 2023 •

edited

Loading

drcca commented Sep 20, 2023 •

edited

Loading