Build issues on an arm32v7 docker container on an AArch64 host

[umarcor]

The following docker image is successfully built on an AArch64 host:

FROM arm64v8/ubuntu:bionic

RUN apt update \
 && DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends \
   ca-certificates \
   cmake \
   doxygen \
   g++-6 \
   gcc \
   git \
   ghostscript \
   imagemagick \
   make \
   python3 \
   transfig \
 && apt autoclean && apt clean && apt -y autoremove \
 && update-ca-certificates \
 && rm /usr/bin/gcc \
 && ln -s /usr/bin/g++-6 /usr/bin/g++ \
 && ln -s /usr/bin/gcc-6 /usr/bin/gcc \
 && git clone https://github.com/DynamoRIO/dynamorio \
 && mkdir build && cd build \
 && cmake ../dynamorio && make -j

$ docker build -t dr/test .

However, if the first line is replaced with FROM arm32v7/ubuntu:bionic, the following errors are shown:

[  1%] Building CXX object clients/drcachesim/CMakeFiles/drmemtrace_simulator.dir/simulator/cache.cpp.o
g++: error: unrecognized command line option '-mpreferred-stack-boundary=2'; did you mean '-mstructure-size-boundary='?
clients/drcpusim/CMakeFiles/drcpusim_ops.dir/build.make:86: recipe for target 'clients/drcpusim/CMakeFiles/drcpusim_ops.dir/options.cpp.o' failed
make[2]: *** [clients/drcpusim/CMakeFiles/drcpusim_ops.dir/options.cpp.o] Error 1
...
[  3%] Building C object tools/CMakeFiles/runstats.dir/runstats.c.o
g++: error: unrecognized command line option '-mpreferred-stack-boundary=2'; did you mean '-mstructure-size-boundary='?
clients/drcachesim/CMakeFiles/drcachesim_ops.dir/build.make:86: recipe for target 'clients/drcachesim/CMakeFiles/drcachesim_ops.dir/common/options.cpp.o' failed
make[2]: *** [clients/drcachesim/CMakeFiles/drcachesim_ops.dir/common/options.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
...
[  7%] Building C object core/CMakeFiles/drhelper.dir/lib/dr_helper.c.o
In file included from /dynamorio/core/arch/asm_shared.asm:37:0:
/dynamorio/core/arch/asm_defines.asm:65:3: error: #error ARM is only 32-bit; AARCH64 is 64-bit
 # error ARM is only 32-bit; AARCH64 is 64-bit

This is likely to be related to DynamoRIO/drmemory#2017. So, I tried replacing the last line in the dockerfile with && linux32 cmake ../dynamorio && make -j:

[ 64%] Building C object core/CMakeFiles/drinjectlib.dir/config.c.o
/dynamorio/core/unix/injector.c:866:50: error: 'PTRACE_GETFPXREGS' undeclared here (not in a function)
                                                { PTRACE_GETFPXREGS, "PTRACE_GETFPXREGS" },
                                                  ^~~~~~~~~~~~~~~~~
/dynamorio/core/unix/injector.c:867:50: error: 'PTRACE_SETFPXREGS' undeclared here (not in a function)
                                                { PTRACE_SETFPXREGS, "PTRACE_SETFPXREGS" },
                                                  ^~~~~~~~~~~~~~~~~
make[2]: *** [core/CMakeFiles/drinjectlib.dir/unix/injector.c.o] Error 1

Than, according to Cross-Compiling for ARM on Linux, I tried cmake -DCMAKE_TOOLCHAIN_FILE=../dynamorio/make/toolchain-arm32.cmake ../dynamorio:

-- The CXX compiler identification is unknown
CMake Error at CMakeLists.txt:105 (enable_language):
  The CMAKE_CXX_COMPILER:

    arm-linux-gnueabihf-g++

  is not a full path and was not found in the PATH.

So, I added ln -s /usr/bin/arm-linux-gnueabihf-g++-6 /usr/bin/arm-linux-gnueabihf-g++ to the dockerfile, and I tried to build it again:

-- Performing Test no_pie_avail - Success
CMake Error at CMakeLists.txt:1624 (message):
  cannot find required libs m, dl, and/or pthread


-- Configuring incomplete, errors occurred!

What is the correct approach to build DynamoRIO on an arm32v7 docker container on an AArch64 host? I would expect the configuration to detect the environtment as armv7 and proceed with a native compilation., since this is what happens with most tools.

[umarcor]

Some further tests:

~$ uname -a
Linux hostname 4.9.0-8-arm64 #1 SMP Debian 4.9.110-3+deb9u5 (2018-09-30) aarch64 GNU/Linux

~$ linux32 uname -a
Linux hostname 4.9.0-8-arm64 #1 SMP Debian 4.9.110-3+deb9u5 (2018-09-30) armv8l GNU/Linux

~$ gcc -dumpmachine
aarch64-linux-gnu

#---

~$ docker run --rm arm64v8/ubuntu:bionic bash -c "uname -a"
Linux bad16824d258 4.9.0-8-arm64 #1 SMP Debian 4.9.110-3+deb9u5 (2018-09-30) aarch64 aarch64 aarch64 GNU/Linux

~$ docker run --rm arm64v8/ubuntu:bionic bash -c "linux32 uname -a"
Linux 52b74bb1d08c 4.9.0-8-arm64 #1 SMP Debian 4.9.110-3+deb9u5 (2018-09-30) armv8l armv8l armv8l GNU/Linux

~$ docker run --rm arm64v8/ubuntu:bionic bash -c "apt update && apt install -y gcc && gcc -dumpmachine"
aarch64-linux-gnu

#---

~$ docker run --rm arm32v7/ubuntu:bionic bash -c "uname -a"
Linux 4783d8df2a45 4.9.0-8-arm64 #1 SMP Debian 4.9.110-3+deb9u5 (2018-09-30) aarch64 aarch64 aarch64 GNU/Linux

~$ docker run --rm arm32v7/ubuntu:bionic bash -c "linux32 uname -a"
Linux b2d6209dcbda 4.9.0-8-arm64 #1 SMP Debian 4.9.110-3+deb9u5 (2018-09-30) armv8l armv8l armv8l GNU/Linux

~$ docker run --rm arm32v7/ubuntu:bionic bash -c "apt update && apt install -y gcc && gcc -dumpmachine"
arm-linux-gnueabihf

[egrimley]

This is likely to be related to DynamoRIO/drmemory#2017<DynamoRIO/drmemory#2017>. So, I tried replacing the last line in the dockerfile with && linux32 cmake ../dynamorio && make -j:

Shouldn't this be "&& linux32 cmake ../dynamiorio && linux32 make -j"? (It's possible that the make stage is also personality-sensitive.) IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

[umarcor]

Shouldn't this be "&& linux32 cmake ../dynamiorio && linux32 make -j"? (It's possible that the make stage is also personality-sensitive.)

Certainly, I think that neither of them should be required. Furthermore, they do not produce the expected result. As shown in my second comment above, linux32 uname -a returns armv8l, but I am expecting armv7, armv7l or armhf, because it is being executed in an arm32v7 docker container.

For the record, I successfully built beehive-lab/mambo on arm32v7 containers on armv8 hosts, with either raspbian (armv6) or debian (aarch64) as host OSs. MAMBO uses -dumpmachine, which returns armhf independently of the host OS.

Anyway, I now tried && linux32 cmake ../dynamiorio && linux32 make -j. The output is the same: ...error: 'PTRACE_GETFPXREGS'....

[egrimley]

Certainly, I think that neither of them should be required.

Yes, ideally, but I'm not sure whose job it should be to set the personality correctly. The Docker daemon's? It would be interesting to take a look at the corresponding situation on Intel. Does Docker change the personality to 32 bits in that case?

Furthermore, they do not produce the expected result. As shown in my second comment above, linux32 uname -a returns armv8l, but I am expecting armv7, armv7l or armhf, because it is being executed in an arm32v7 docker container.

I don't think the Linux personality can be expected to degrade the architecture version like that: with "linux32" you switch into Armv8's 32-bit state, not into Armv7, so I think "armv8l" is correct, and I think I've seen it working with "armv8l" in a chroot (rather than a Docker container). I think it would be worth investigating the PTRACE_GETFPXREGS problem a bit more. It might just be that you need another set of header files installed, or your tool chain is too modern, or something else not related to 32/64 bits. It sounds somehow familiar to me, but I might be mixing it up with a different ptrace-related thing I once had to patch. Edmund IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

[umarcor]

Certainly, I think that neither of them should be required.
Yes, ideally, but I'm not sure whose job it should be to set the personality correctly. The Docker daemon's?

AFAIK, docker uses the kernel on the host, which is an armv8. Therefore, it is ok that queries to the kernel return aarch64 or armv8l, since those are the architectures supported by the host.

It would be interesting to take a look at the corresponding situation on Intel. Does Docker change the personality to 32 bits in that case?

The behaviour is the same. E.g., on a Win10 host:

# winpty docker run --rm -it i386/ubuntu:bionic bash
root@d262ae1c99fa:/# uname -a
Linux d262ae1c99fa 4.9.125-linuxkit #1 SMP Fri Sep 7 08:20:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@d262ae1c99fa:/# linux32 uname -a
Linux d262ae1c99fa 4.9.125-linuxkit #1 SMP Fri Sep 7 08:20:28 UTC 2018 i686 i686 i686 GNU/Linux

# winpty docker run --rm -it ubuntu:bionic bash
root@e80195e855f2:/# uname -a
Linux e80195e855f2 4.9.125-linuxkit #1 SMP Fri Sep 7 08:20:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@e80195e855f2:/# linux32 uname -a
Linux e80195e855f2 4.9.125-linuxkit #1 SMP Fri Sep 7 08:20:28 UTC 2018 i686 i686 i686 GNU/Linux

I don't think the Linux personality can be expected to degrade the architecture version like that: with "linux32" you switch into Armv8's 32-bit state, not into Armv7, so I think "armv8l" is correct, and I think I've seen it working with "armv8l" in a chroot (rather than a Docker container).

I think I didn't word it correctly. I am not expecting uname -a to return anything different than aarch64 or armv8l. I meant that uname (which I think that DynamoRIO uses) seems not to be a valid mechanism in this context, because forcing it to degrade the architecture version is ugly.

IMHO, using linux32 is irrelevant, because armv8l is not suitable in this context. Although the image based on arm32v7 is being built on an armv8 host, it is then to be pushed to docker hub. Any user with an armv7 host should be able to use it. This is not guaranteed if features available in armv8l but not in armv7 are used.

NOTE: I am considering armv7 == armhf, armv8l == aarch32 and armv7 ⊂ armv8l, but armv7 != armv8l.

---

I think it would be worth investigating the PTRACE_GETFPXREGS problem a bit more. It might just be that you need another set of header files installed, or your tool chain is too modern, or something else not related to 32/64 bits. It sounds somehow familiar to me, but I might be mixing it up with a different ptrace-related thing I once had to patch.

I am using GCC 6, because the build would fail with either GCC 7 (on aarch64) or GCC 8 (on x86-64).

I will investigate the PTRACE_GETFPXREGS problem. Nonetheless, as commented above, this will be a fix for armv8l. I think that I will still need some mechanism to tell cmake that the target is armv7. Note, that I need to do this while avoiding arm-linux-gnueabihf- to be prepended to all the resources.

[umarcor]

I will investigate the PTRACE_GETFPXREGS problem.

I found a workaround. After commenting core/unix/injector.c#L865-L868, linux32 cmake ../dynamorio/ && make -j is successfully executed. The second linux32 is not required.

Therefore, it seems that ifndef AARCH64 might be changed to something like ifndef ARMV8.

[umarcor]

I tried to build DynamoRIO natively on a PYNQ board. I got the same PTRACE_GETFPXREGS error. So, I applied the patch in #3382. Summarizing, now DynamoRIO is successfully built:

• natively on armv7l
• for aarch64 on an arm64v8 container on an armv8 host
• for armv8l/aarch32 on an arm32v7l container on an armv8 host.

Therefore, the use case of this issue is still unresolved: how to build for armv7l/armhf on an arm32v7 container on an armv8 host.

EDIT

This is the output when I try to execute the binaries built inside a container (for armv8l) on the PYNQ:

 ./bin32/drrun /bin/ls
ERROR: unable to inject: exec of |(null)| failed

[egrimley]

This worked for me (with 13 test failures):

FROM arm32v7/debian:stretch
RUN apt update
RUN apt-get dist-upgrade -y
RUN apt-get install -y --no-install-recommends ca-certificates cmake g++ git make
WORKDIR /tmp
RUN git clone https://github.com/DynamoRIO/dynamorio
RUN mkdir /build
WORKDIR build
RUN linux32 cmake -DBUILD_TESTS=ON ../dynamorio
RUN linux32 make -j4
RUN linux32 make test

There are problems with debian:buster or ubuntu:bionic, as you have observed. Have you tested the same recipe on Intel to confirm that this problem is Arm-specific?

As mentioned elsewhere, if you want to build with GCC 7, replace || with | in line 1888 of opnd_shared.c: the compiler warning seems to have identified a real bug.

[umarcor]

Hi @egrimley! I can successfully build DynamoRIO in multiple contexts and with different versions of GCC. As commented above, this issue is not about building DynamoRIO, but about how to tell cmake which architecture to build for. Precisely:

• [Expected] Built natively on an armv7l (ZYNQ with Ubuntu 18.04), the artifact works on armv7l/armhf.
• [Expected] Built on an arm64v8 container on an armv8 host, the artifact works on aarch64.
• [Expected] Built on an arm64v8 container on an armv8 host, prepending linux32 to cmake, the artifact works on armv8l.
• [Not expected] Built on an arm32v7 container on an armv8 host, the artifact works on aarch64.
• [Not expected] Built on an arm32v7 container on an armv8 host, prepending linux32 to cmake, the artifact works on armv8l.

The last case is not consistent with how make behaves with other applications such as MAMBO or GHDL. How can I build DynamoRIO for armv7l/armhf on an arm32v7 container on an armv8 host?

You can try what I mean by pushing the image you built (based on arm32v7/debian:stretch) to dockerhub. Then just pull it from any armv7 device and try to execute it. If you don't have any armv7 device with docker, just copy /build from the image to the target device. Because DynamoRIO was built for armv8, indeed, it will fail. So, it is not an arm32v7 image anymore, but some mix between it and arm32v8 (which does not exist).

---

Don't take me wrong, I am interested on finding the bugs with different versions of the compiler on different platforms. But I wouldn't like this issue to focus on that. It is just casual that I stumbled upon those while trying different alternatives.

---

There are problems with debian:buster or ubuntu:bionic, as you have observed. Have you tested the same recipe on Intel to confirm that this problem is Arm-specific?

On the one hand, I don't need to use DynamoRIO on neither i686 nor i386. On the other hand, I don't have any device which can only execute i386, so I cannot test it. But, if I had it, I believe that the result would be the same because (as commented above):

# winpty docker run --rm -it i386/ubuntu:bionic bash -c "uname -m"
x86_64

# winpty docker run --rm -it i386/ubuntu:bionic bash -c "linux32 uname -m"
i686

# winpty docker run --rm -it i386/ubuntu:bionic bash -c "apt update; apt-get install gcc; gcc -dumpmachine"
(...)
i686-linux-gnu

# winpty docker run --rm -it i386/ubuntu:bionic bash -c "apt update; apt-get install clang; clang -dumpmachine"
(...)
i686-pc-linux-gnu

[egrimley]

There are these lines in CMakeLists.txt:

# The target architecture.
# For cross-compilation this should still work as you're supposed to set this var.
if (CMAKE_SYSTEM_PROCESSOR MATCHES "^arm")

So perhaps one should be running cmake -DCMAKE_SYSTEM_PROCESSOR=arm ...

There is plenty of online discussion of "cmake detect architecture", for example: https://stackoverflow.com/questions/11944060/how-to-detect-target-architecture-using-cmake

[umarcor]

So perhaps one should be running cmake -DCMAKE_SYSTEM_PROCESSOR=arm ...

Thanks! That's a great hint. Unfortunately, it does not work because CMAKE_SYSTEM_PROCESSOR is overwritten in CMakeLists.txt. Precisely, I have found it to change in line CMakeLists.txt#L94. I am investigating it.

There is plenty of online discussion of "cmake detect architecture", for example: https://stackoverflow.com/questions/11944060/how-to-detect-target-architecture-using-cmake

Since the use case is quite specific, I am ok with setting the processor/arch explicitly. So, it is not necessary for cmake to support it.

EDIT

Although CMAKE_SYSTEM_PROCESSOR alone does not work, if used along with CMAKE_SYSTEM_NAME it seems to work: cmake -DCMAKE_SYSTEM_NAME=Linux -DCMAKE_SYSTEM_PROCESSOR=arm ../dynamorio/. CMAKE_SYSTEM_VERSION seems not to be required.

I will copy the artifacts to a PYNQ board later, in order to verify that the target is armv7 and not armv8.

• https://gitlab.kitware.com/cmake/cmake/issues/16909
• https://cmake.org/cmake/help/v3.4/variable/CMAKE_SYSTEM_NAME.html
• https://cmake.org/cmake/help/v3.4/variable/CMAKE_SYSTEM_VERSION.html

[umarcor]

I have now tested it and it works. I can build the following docker image on an armv8 host, which can then be pushed to dockerhub and pulled to armv7 devices.

FROM arm32v7/ubuntu:bionic
RUN apt update
RUN apt-get install -y --no-install-recommends ca-certificates cmake g++-6 gcc git make python
RUN rm -rf /usr/bin/gcc \
 && ln -s /usr/bin/gcc-6 /usr/bin/gcc \
 && ln -s /usr/bin/g++-6 /usr/bin/g++ \
 && git clone -b fix-ptrace-fpxregs https://github.com/umarcor/dynamorio \
 && cd dynamorio && mkdir build && cd build \
 && cmake -DCMAKE_SYSTEM_NAME=Linux -DCMAKE_SYSTEM_PROCESSOR=arm .. \
 && make -j4

Notes:

• -DCMAKE_SYSTEM_PROCESSOR=armv7 is also supported, and it is a safer value, indeed.
• Branch fix-ptrace-fpxregs is Do not reference PTRACE_{G,S}ETFPXREGS when undefined #3382.
• g++-6 is required because of i#2901: Fix undefined L_aeabi_ldivmod #3403.
• As soon as those fixes are merged to master, the following Dockerfile will also work:

FROM arm32v7/ubuntu:bionic
RUN apt update && apt-get install -y --no-install-recommends ca-certificates cmake g++ gcc git make python
RUN git clone https://github.com/DynamoRIO/dynamorio \
 && mkdir dynamorio/build && cd dynamorio/build \
 && cmake -DCMAKE_SYSTEM_NAME=Linux -DCMAKE_SYSTEM_PROCESSOR=armv7 .. \
 && make -j4

---

I'm closing this because the issue about how to properly set the target processor is already solved and tested.

Thanks a lot @egrimley!

[umarcor]

Just an additional note. Setting CMAKE_SYSTEM_NAME and CMAKE_SYSTEM_PROCESSOR seems not to be required if the ARM docker containers are executed on x86_64 through qemu-user-static binaries (either arm or arrch64).