Is this a cmake issue or QEMU issue? Broken compile on armhf, works on Aarch64 under Buildx/QEMU #107

rhastie · 2020-04-28T17:20:53Z

Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)

/kind question

Possible issue or bug with QEMU - need help identifying where issue is

Description:

I'm using latest multiarch/qemu-user-static installed via Docker container.

I run some builds using QEMUs via Docker buildx. Docker builds work fine for AArch64, but I consistently get a failure like below when attempting to build for 32-bit ARM (hard-float).

The containers are successfully built for AMD64 (ie. with no emulation) and Aarch64 (using QEMU emulation), but, fail for the ARMHF (ARMv7 32-bit) build again using QEMU emulation. I am using Focal 20.04 as the host and also building a Focal 20.04 container.

I have specifically used cmake 3.16.3 (from Focal 20.04 repo) and also compiled and used cmake 3.17.1

For reference, I am also using Conan as the C++ package manager - Version 1.24.1 / Docker 19.03.8 with the very latest buildx / Moby-Buildkit v0.7.1... All I believe are current

Please find attached the output of my failure

buildx_armhf_failure.txt

Finally, I have done my best to enable the--trace-expand option on cmake and redirect stdout and stderr - As you can see its not perfect but I'm hoping it will give you something to go on..

stdout.out.txt
stderr.out (1).txt

Steps to reproduce the issue:

Generally following this guide using multiarch/qemu-user-static - https://medium.com/@artur.klauser/building-multi-architecture-docker-images-with-buildx-27d80f7e2408
I can provide access to a Git containing the build Dockerfile - Please let me know the required Git account
Typically setting buildx to support linux ARM/v7, Aarch64 and amd64
Run using: docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7 -t rhastie/container .

Describe the results you received:

Container is successfully built for amd64 (ie. x86_64) no QEMU emulation
Container is successfully built for Aarch64 via QEMU emulation
Container build errors and terminates for Arm/v7 via QEMU emulation

Describe the results you expected:

I would expect the container to be built for both architectures - This may be either cmake issue or a QEMU issue. Unusual that it works for Aarch64 and not armhf under QEMU.

Environment:

x86-64 Running Ubuntu Focal 20.04
Docker 19.03.8
Moby-Buildkit v0.7.1
cmake 3.16.3 and cmake 3.17.1 tested
latest buildx CLI from - https://github.com/docker/buildx
latest QEMU-user-static from - https://github.com/multiarch/qemu-user-static
Container being built using Focal 20.04

QEMU version: (if you can know it):
Container application: Docker
Output of docker version, podman version or singularity version

Client:
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.13.8
 Git commit:        afacb8b7f0
 Built:             Wed Mar 11 23:42:35 2020
 OS/Arch:           linux/amd64
 Experimental:      true

Server:
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       afacb8b7f0
  Built:            Wed Mar 11 22:48:33 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.3-0ubuntu2
  GitCommit:        
 runc:
  Version:          spec: 1.0.1-dev
  GitCommit:        
 docker-init:
  Version:          0.18.0
  GitCommit:

Additional information optionally:
I also have an issue raised here: https://gitlab.kitware.com/cmake/cmake/-/issues/20568

The text was updated successfully, but these errors were encountered:

junaruga · 2020-04-28T17:50:52Z

I'm using latest multiarch/qemu-user-static installed via Docker container.

Are you sure? This repository is for "multiarch/qemu-user-static" container image.

I run some builds using QEMUs via Docker buildx.

In my memory docker buildx has using the forked version of qemu internally. This qemu. Not original one.
https://github.com/moby/qemu

Generally following this guide using multiarch/qemu-user-static - https://medium.com/@artur.klauser/building-multi-architecture-docker-images-with-buildx-27d80f7e2408

The used "qemu-user-static" is not this repository's one.

$ sudo apt-get install -y qemu-user-static

But for example if you are running Ubuntu as host bionic, here is the "qemu-user-static" deb package.
https://packages.ubuntu.com/bionic-updates/qemu-user-static

If you have a question related to buildx, the place to ask is here.
https://github.com/docker/buildx

rhastie · 2020-04-28T17:56:30Z

Yes... definitely using "multiarch/qemu-user-static" container image

Search inside the article (https://medium.com/@artur.klauser/building-multi-architecture-docker-images-with-buildx-27d80f7e2408) for "Docker Image Based Installation" - You'll see the references to multiarch/qemu-user-static. I'm using this approach, however it is confusing as several approaches are described.

rhastie · 2020-05-23T19:11:08Z

I think I've found the culprit... Building using bionic means a glibc version < 2.28, Focal is >2.28 - It appears the READDIR is returning NULL

https://bugs.launchpad.net/qemu/+bug/1805913

Is there any easy way round this bug/issue?

junaruga · 2020-05-23T19:27:01Z

I still do not understand your steps.

For Steps to reproduce the issue:, could you describe the steps by only command lines and result like this?

$ command1
result1

$ command2
result2

$ command3
result3

...

And I can not access to your log files. I do not know why.

stdout.out.txt
stderr.out (1).txt

Could you upload the log to your https://gist.github.com ?

junaruga · 2020-05-23T19:34:18Z

And it's good if you share your the minimal Dockerfile to reproduce the issue too.

junaruga · 2020-05-23T19:46:49Z

https://medium.com/@artur.klauser/building-multi-architecture-docker-images-with-buildx-27d80f7e2408

Docker Image Based Installation
As an alternative to installing the QEMU and binfmt-support packages on your host system you can use a docker image to satisfy the corresponding requirements. There are several docker images that do the job, among them multiarch/qemu-user-static and docker/binfmt.

If you faced the error as a result of docker buildx ..., in my experience, multiarch/qemu-user-statuc is not related to the result. Because docker buildx uses the internal binfmt_misc and qemu. I think the referred article might be wrong.

If you executed the multiarch/qemu-user-static image, possibly you see /proc/sys/fs/binfmt_misc/qemu-$arch files on your host. So, it might be good to remove the files at once.

https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html

You can remove one entry or all entries by echoing -1 to /proc/.../the_name or /proc/sys/fs/binfmt_misc/status.

# echo -1 > /proc/sys/fs/binfmt_misc/qemu-$arch

Please do it on your responsibility.

rhastie · 2020-05-23T19:54:11Z

Junaruga,

Gist is here: https://gist.github.com/rhastie/04e5af6fa4f61e8e278a1a5ee27484f2

It is quite an involved Dockerfile inside a private repo which is using cmake to run a compile - If you share your GitHub ID I can add you to the repo - Focal version which is erroring is on the focal branch. Currently master is Bionic - I'm having to revert to Bionic.

All I am changing between the two Dockerfile versions is the base-image

junaruga · 2020-05-23T20:08:57Z

Now I can see stderr.out.1.txt. Thank you.
Is the log the output of docker buildx ..., right?

I do not want to see your private repository. It's okay not to share it.

This CI service can offer ARM 32 bit native host environment. It's free for public repository. It might not free for the private repository. But it's worth to try.
https://drone.io/

junaruga · 2020-05-23T20:10:15Z

The example.
https://github.com/junaruga/ci-multi-arch-test/blob/master/.drone.yml#L27

rhastie · 2020-05-23T20:18:21Z

Now I can see stderr.out.1.txt. Thank you.
Is the log the output of docker buildx ..., right?

I do not want to see your private repository. It's okay not to share it.

This CI service can offer ARM 32 bit native host environment. It's free for public repository. It might not free for the private repository. But it's worth to try.
https://drone.io/

Yes - This is part of the docker buildx output for just ARMv7 build

Thanks for the help - I'll look at https://drone.io

riku76 · 2020-08-20T06:52:25Z

Just wanted to comment that I have encountered this same problem and it seems to be tied to cmake version. 3.13.4 works fine, but cmake 3.18.1 fails to exact same error in the same container.

TheBrokenRail · 2020-09-21T14:31:44Z

This is reported on CMake's GitLab here: https://gitlab.kitware.com/cmake/cmake/-/issues/20568

umarcor · 2020-10-26T06:31:51Z

See https://github.com/dbhi/docker#building-docker-images-foron-arm-targets:

Building DynamoRIO on an arm32v7 container on a 64 bit host requires to set -DCMAKE_SYSTEM_NAME=Linux -DCMAKE_SYSTEM_PROCESSOR=armv7, as explained at DynamoRIO/dynamorio#3381.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Is this a cmake issue or QEMU issue? Broken compile on armhf, works on Aarch64 under Buildx/QEMU #107

Is this a cmake issue or QEMU issue? Broken compile on armhf, works on Aarch64 under Buildx/QEMU #107

rhastie commented Apr 28, 2020 •

edited

Loading

junaruga commented Apr 28, 2020

rhastie commented Apr 28, 2020 •

edited

Loading

rhastie commented May 23, 2020

junaruga commented May 23, 2020 •

edited

Loading

junaruga commented May 23, 2020

junaruga commented May 23, 2020 •

edited

Loading

rhastie commented May 23, 2020

junaruga commented May 23, 2020

junaruga commented May 23, 2020

rhastie commented May 23, 2020

riku76 commented Aug 20, 2020

TheBrokenRail commented Sep 21, 2020

umarcor commented Oct 26, 2020

Is this a cmake issue or QEMU issue? Broken compile on armhf, works on Aarch64 under Buildx/QEMU #107

Is this a cmake issue or QEMU issue? Broken compile on armhf, works on Aarch64 under Buildx/QEMU #107

Comments

rhastie commented Apr 28, 2020 • edited Loading

junaruga commented Apr 28, 2020

rhastie commented Apr 28, 2020 • edited Loading

rhastie commented May 23, 2020

junaruga commented May 23, 2020 • edited Loading

junaruga commented May 23, 2020

junaruga commented May 23, 2020 • edited Loading

rhastie commented May 23, 2020

junaruga commented May 23, 2020

junaruga commented May 23, 2020

rhastie commented May 23, 2020

riku76 commented Aug 20, 2020

TheBrokenRail commented Sep 21, 2020

umarcor commented Oct 26, 2020

rhastie commented Apr 28, 2020 •

edited

Loading

rhastie commented Apr 28, 2020 •

edited

Loading

junaruga commented May 23, 2020 •

edited

Loading

junaruga commented May 23, 2020 •

edited

Loading