[Bug]: clang crashes on aarch64 architecture under x86_64 host when using qemu-static

[licy183]

Problem description

clang crashes/hangs on aarch64 architecture under x86_64 host. It seems to be an upstream issue of qemu-aarch64-static. But actually there is no such issue under other aarch64 rootfs (test with clang under alpine and ubuntu).

What steps will reproduce the bug?

pkg install clang
echo -e "#include <iostream>\nint main() { std::cout << std::endl; }" > main.cpp
clang++ main.cpp -v # hangs
clang++  -std=c++11 main.cpp -v # crashes

Log

clang version 14.0.1
Target: aarch64-unknown-linux-android24
Thread model: posix
InstalledDir: /data/data/com.termux/files/usr/bin
 "/data/data/com.termux/files/usr/bin/clang-14" -cc1 -triple aarch64-unknown-linux-android24 -emit-obj -mrelax-all --mrelax-relocations -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name main.cpp -mrelocation-model pic -pic-level 2 -pic-is-pie -mframe-pointer=non-leaf -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu generic -target-feature +neon -target-feature +v8a -target-feature +fix-cortex-a53-835769 -target-abi aapcs -fallow-half-arguments-and-returns -mllvm -treat-scalable-fixed-error-as-warning -debugger-tuning=gdb -v -fcoverage-compilation-dir=/data/data/com.termux/files/home -resource-dir /data/data/com.termux/files/usr/lib/clang/14.0.1 -isysroot /data/data/com.termux/files -internal-isystem /data/data/com.termux/files/usr/include/c++/v1 -internal-isystem /data/data/com.termux/files/usr/lib/clang/14.0.1/include -internal-isystem /data/data/com.termux/files/usr/local/include -internal-externc-isystem /data/data/com.termux/files/usr/include/aarch64-linux-android -internal-externc-isystem /data/data/com.termux/files/include -internal-externc-isystem /data/data/com.termux/files/usr/include -std=c++11 -fdeprecated-macro -fdebug-compilation-dir=/data/data/com.termux/files/home -ferror-limit 19 -fno-signed-char -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -target-feature +outline-atomics -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /data/data/com.termux/files/usr/tmp/main-1c1e58.o -x c++ main.cpp
clang -cc1 version 14.0.1 based upon LLVM 14.0.1 default target aarch64-unknown-linux-android24
ignoring nonexistent directory "/data/data/com.termux/files/usr/local/include"
ignoring nonexistent directory "/data/data/com.termux/files/include"
#include "..." search starts here:
#include <...> search starts here:
 /data/data/com.termux/files/usr/include/c++/v1
 /data/data/com.termux/files/usr/lib/clang/14.0.1/include
 /data/data/com.termux/files/usr/include/aarch64-linux-android
 /data/data/com.termux/files/usr/include
End of search list.
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: /data/data/com.termux/files/usr/bin/clang-14 -cc1 -triple aarch64-unknown-linux-android24 -emit-obj -mrelax-all --mrelax-relocations -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name main.cpp -mrelocation-model pic -pic-level 2 -pic-is-pie -mframe-pointer=non-leaf -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu generic -target-feature +neon -target-feature +v8a -target-feature +fix-cortex-a53-835769 -target-abi aapcs -fallow-half-arguments-and-returns -mllvm -treat-scalable-fixed-error-as-warning -debugger-tuning=gdb -v -fcoverage-compilation-dir=/data/data/com.termux/files/home -resource-dir /data/data/com.termux/files/usr/lib/clang/14.0.1 -isysroot /data/data/com.termux/files -internal-isystem /data/data/com.termux/files/usr/include/c++/v1 -internal-isystem /data/data/com.termux/files/usr/lib/clang/14.0.1/include -internal-isystem /data/data/com.termux/files/usr/local/include -internal-externc-isystem /data/data/com.termux/files/usr/include/aarch64-linux-android -internal-externc-isystem /data/data/com.termux/files/include -internal-externc-isystem /data/data/com.termux/files/usr/include -std=c++11 -fdeprecated-macro -fdebug-compilation-dir=/data/data/com.termux/files/home -ferror-limit 19 -fno-signed-char -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -target-feature +outline-atomics -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /data/data/com.termux/files/usr/tmp/main-1c1e58.o -x c++ main.cpp
1.      /data/data/com.termux/files/usr/include/c++/v1/algorithm:3123:1: current parser token '{'
2.      /data/data/com.termux/files/usr/include/c++/v1/algorithm:660:1 <Spelling=/data/data/com.termux/files/usr/include/c++/v1/__config:793:37>: parsing namespace 'std'
3.      /data/data/com.termux/files/usr/include/c++/v1/algorithm:660:1 <Spelling=/data/data/com.termux/files/usr/include/c++/v1/__config:793:60>: parsing namespace 'std::__ndk1'
libc: Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xfffffff0 in tid 980 (clang-14), pid 980 (clang-14)
libc: failed to spawn debuggerd dispatch thread: Invalid argument

[Grimler91]

Does it only happen in c++ mode?

[licy183]

Does it only happen in c++ mode?

Yes, .c code compiles successfully.

[licy183]

I ran the termux-docker:aarch64 rootfs using proot under a termux-docker:x86_64 container, and got another error as following.

~ $ cat main.cpp 
#include <iostream>
int main() {
    std::cout << std::endl;
}
~ $ clang main.cpp -v
clang version 14.0.3
Target: aarch64-unknown-linux-android24
Thread model: posix
InstalledDir: /data/data/com.termux/files/usr/bin
 "/data/data/com.termux/files/usr/bin/clang-14" -cc1 -triple aarch64-unknown-linux-android24 -emit-obj -mrelax-all --mrelax-relocations -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name main.cpp -mrelocation-model pic -pic-level 2 -pic-is-pie -mframe-pointer=non-leaf -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu generic -target-feature +neon -target-feature +v8a -target-feature +fix-cortex-a53-835769 -target-abi aapcs -fallow-half-arguments-and-returns -mllvm -treat-scalable-fixed-error-as-warning -debugger-tuning=gdb -v -fcoverage-compilation-dir=/data/data/com.termux/files/home -resource-dir /data/data/com.termux/files/usr/lib/clang/14.0.3 -isysroot /data/data/com.termux/files -internal-isystem /data/data/com.termux/files/usr/include/c++/v1 -internal-isystem /data/data/com.termux/files/usr/lib/clang/14.0.3/include -internal-isystem /data/data/com.termux/files/usr/local/include -internal-externc-isystem /data/data/com.termux/files/usr/include/aarch64-linux-android -internal-externc-isystem /data/data/com.termux/files/include -internal-externc-isystem /data/data/com.termux/files/usr/include -fdeprecated-macro -fdebug-compilation-dir=/data/data/com.termux/files/home -ferror-limit 19 -fno-signed-char -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -target-feature +outline-atomics -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /data/data/com.termux/files/usr/tmp/main-3ce7f8.o -x c++ main.cpp
clang -cc1 version 14.0.3 based upon LLVM 14.0.3 default target aarch64-unknown-linux-android24
ignoring nonexistent directory "/data/data/com.termux/files/usr/local/include"
ignoring nonexistent directory "/data/data/com.termux/files/include"
#include "..." search starts here:
#include <...> search starts here:
 /data/data/com.termux/files/usr/include/c++/v1
 /data/data/com.termux/files/usr/lib/clang/14.0.3/include
 /data/data/com.termux/files/usr/include/aarch64-linux-android
 /data/data/com.termux/files/usr/include
End of search list.
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: /data/data/com.termux/files/usr/bin/clang-14 -cc1 -triple aarch64-unknown-linux-android24 -emit-obj -mrelax-all --mrelax-relocations -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name main.cpp -mrelocation-model pic -pic-level 2 -pic-is-pie -mframe-pointer=non-leaf -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu generic -target-feature +neon -target-feature +v8a -target-feature +fix-cortex-a53-835769 -target-abi aapcs -fallow-half-arguments-and-returns -mllvm -treat-scalable-fixed-error-as-warning -debugger-tuning=gdb -v -fcoverage-compilation-dir=/data/data/com.termux/files/home -resource-dir /data/data/com.termux/files/usr/lib/clang/14.0.3 -isysroot /data/data/com.termux/files -internal-isystem /data/data/com.termux/files/usr/include/c++/v1 -internal-isystem /data/data/com.termux/files/usr/lib/clang/14.0.3/include -internal-isystem /data/data/com.termux/files/usr/local/include -internal-externc-isystem /data/data/com.termux/files/usr/include/aarch64-linux-android -internal-externc-isystem /data/data/com.termux/files/include -internal-externc-isystem /data/data/com.termux/files/usr/include -fdeprecated-macro -fdebug-compilation-dir=/data/data/com.termux/files/home -ferror-limit 19 -fno-signed-char -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -target-feature +outline-atomics -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /data/data/com.termux/files/usr/tmp/main-3ce7f8.o -x c++ main.cpp
1.      /data/data/com.termux/files/usr/include/c++/v1/__locale:1269:1 <Spelling=/data/data/com.termux/files/usr/include/c++/v1/__config:914:66>: current parser token ';'
2.      /data/data/com.termux/files/usr/include/c++/v1/__locale:49:1 <Spelling=/data/data/com.termux/files/usr/include/c++/v1/__config:793:37>: parsing namespace 'std'
3.      /data/data/com.termux/files/usr/include/c++/v1/__locale:49:1 <Spelling=/data/data/com.termux/files/usr/include/c++/v1/__config:793:60>: parsing namespace 'std::__ndk1'
4.      /data/data/com.termux/files/usr/include/c++/v1/__locale:1269:1 <Spelling=/data/data/com.termux/files/usr/include/c++/v1/__locale:1269:65>: instantiating class definition 'std::codecvt_byname<char32_t, char, mbstate_t>'
libc: failed to connect to tombstoned: No such file or directory
/home/builder/.termux-build/qemu-system-x86-64-headless/src/include/qemu/rcu.h:102: void rcu_read_unlock(void): assertion "p_rcu_reader->depth != 0" failed
libc: /home/builder/.termux-build/qemu-system-x86-64-headless/src/include/qemu/rcu.h:102: void rcu_read_unlock(void): assertion "p_rcu_reader->depth != 0" failed
clang-14: error: unable to execute command: Aborted (core dumped)
clang-14: error: clang frontend command failed due to signal (use -v to see invocation)
clang version 14.0.3
Target: aarch64-unknown-linux-android24
Thread model: posix
InstalledDir: /data/data/com.termux/files/usr/bin
clang-14: note: diagnostic msg: 
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang-14: note: diagnostic msg: /data/data/com.termux/files/usr/tmp/main-3cd96b.cpp
clang-14: note: diagnostic msg: /data/data/com.termux/files/usr/tmp/main-3cd96b.sh
clang-14: note: diagnostic msg: 

********************

Steps

docker run -td --name=temp termux/termux-docker:aarch64
docker export temp > aarch64-rootfs.tar
docker kill temp
docker rm temp 
docker run --rm -it -v $(pwd)/aarch64-rootfs.tar:/data/data/com.termux/files/home/aarch64-rootfs.tar termux/termux-docker:x86_64
mkdir -p rootfs
pushd rootfs
tar -vxf ../aarch64-rootfs.tar
popd
pkg install qemu-user-aarch64 proot
LD_PRELOAD="" proot -q qemu-aarch64 -R rootfs/ bash
pkg install clang
echo -e "#include <iostream>\nint main() { std::cout << std::endl; }" > main.cpp
clang main.cpp -v

[licy183]

When using libhoudini (rather than qemu-aarch64/arm-static) as the ARM translation layer, clang++ main.cpp seems to work fine. So I think it is an upstream issue of qemu-user-static.

[licy183]

How to use libhoudini with docker ? Any tuts ?

Just like qemu. Register the binfmt_misc and build a custom docker image.

#!/usr/bin/env sh
echo ":arm_dyn:M::\x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x03\x00\x28::/system/bin/houdini:P" | sudo tee /proc/sys/fs/binfmt_misc/register
echo ":arm_exe:M::\x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28::/system/bin/houdini:P" | sudo tee /proc/sys/fs/binfmt_misc/register
echo ":arm64_dyn:M::\x7f\x45\x4c\x46\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x03\x00\xb7::/system/bin/houdini64:P" | sudo tee /proc/sys/fs/binfmt_misc/register
echo ":arm64_exe:M::\x7f\x45\x4c\x46\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7::/system/bin/houdini64:P" | sudo tee /proc/sys/fs/binfmt_misc/register

This is a custom docker image using libhoudini. The libhoudini binaries are from https://github.com/natsumerinchan/libhoudini-package. The following is Dockerfile.

##############################################################################
# Bootstrap Termux environment.
FROM termux/termux-docker:x86_64 AS bootstrap

ARG TARGET_ARCH=aarch64

USER 0:0
SHELL ["/system/bin/sh", "-c"]
ENV PATH /system/bin

# Add libhoudini
ADD https://www.7-zip.org/a/7z2107-linux-x64.tar.xz /7zip.tar.xz
ADD https://github.com/natsumerinchan/libhoudini-package/releases/download/v2.1/Libhoudini_v2.1_natsumerinchan.gxp /libhoudini.gxp
RUN cd /data/data/com.termux/files/usr/tmp && \
    /data/data/com.termux/files/usr/bin/xz -d /7zip.tar.xz && \
    busybox tar xvf /7zip.tar && \
    busybox mkdir ./libhoudini && \
    ./7zzs x /libhoudini.gxp -o./libhoudini -aoa && \
    busybox rm /libhoudini.gxp && \
    busybox cp -r ./libhoudini/system/* /system && \
    cd / && \
    busybox rm -r /data /usr /bin /tmp /7zip.tar

# Extract bootstrap archive and create symlinks.
ADD https://github.com/termux/termux-packages/releases/download/bootstrap-2022.05.01-r1%2Bapt-android-7/bootstrap-$TARGET_ARCH.zip /bootstrap.zip
RUN busybox mkdir -p /data/data/com.termux/files && \
    cd /data/data/com.termux/files && \
    busybox mkdir -p ../cache ./usr ./home && \
    busybox unzip -d usr /bootstrap.zip && \
    busybox rm /bootstrap.zip && \
    cd ./usr && \
    busybox cat SYMLINKS.txt | while read -r line; do \
      dest=$(echo "$line" | busybox awk -F '←' '{ print $1 }'); \
      link=$(echo "$line" | busybox awk -F '←' '{ print $2 }'); \
      busybox ln -s "$dest" "$link"; \
    done && \
    busybox rm SYMLINKS.txt && \
    busybox ln -s /data/data/com.termux/files/usr /usr && \
    busybox ln -s /data/data/com.termux/files/usr/bin /bin && \
    busybox ln -s /data/data/com.termux/files/usr/tmp /tmp

# Set ownership and file access modes:
# * User content is owned by 1000:1000.
# * Termux file modes are set only for user.
# * Rest is owned by root and has 755/644 modes.
RUN busybox chown -Rh 0:0 /system && \
    busybox chown -Rh 1000:1000 /data/data/com.termux && \
    busybox chown 1000:1000 /system/etc/hosts /system/etc/static-dns-hosts.txt && \
    busybox find /system -type d -exec busybox chmod 755 "{}" \; && \
    busybox find /system -type f -executable -exec busybox chmod 755 "{}" \; && \
    busybox find /system -type f ! -executable -exec busybox chmod 644 "{}" \; && \
    busybox find /data -type d -exec busybox chmod 755 "{}" \; && \
    busybox find /data/data/com.termux/files -type f -o -type d -exec busybox chmod g-rwx,o-rwx "{}" \; && \
    cd /data/data/com.termux/files/usr && \
    busybox find ./bin ./lib/apt ./lib/bash ./libexec -type f -exec busybox chmod 700 "{}" \;

# Chmod for libhoudini linker
RUN busybox chmod +x /system/bin/houdini && \
    busybox chmod +x /system/bin/houdini64 && \
    busybox chmod +x /system/bin/arm/linker && \
    busybox chmod +x /system/bin/arm64/linker64

# Add symlinks to $PREFIX/lib
RUN if [ $TARGET_ARCH == 'aarch64' ]; then LIBPREFIX=lib64/arm64; else LIBPREFIX=lib/arm ;fi && \
    for lib in libc++.so libc.so libdl.so libm.so liblog.so; do \
        busybox ln -sf /system/$LIBPREFIX/$lib /data/data/com.termux/files/usr/lib/; \
    done
# Switch user to non-root.
USER 1000:1000

# Update static DNS cache on login. Also symlink script and host list to prefix.
RUN echo "echo -e 'Updating static DNS:\n' && /system/bin/update-static-dns && echo" \
    > /data/data/com.termux/files/home/.bashrc && \
    busybox ln -s /system/bin/update-static-dns /data/data/com.termux/files/usr/bin/update-static-dns && \
    busybox ln -s /system/etc/static-dns-hosts.txt /data/data/com.termux/files/usr/etc/static-dns-hosts.txt

##############################################################################
# Create final image.
FROM scratch

ENV ANDROID_DATA     /data
ENV ANDROID_ROOT     /system
ENV HOME             /data/data/com.termux/files/home
ENV LANG             en_US.UTF-8
ENV PATH             /data/data/com.termux/files/usr/bin
ENV PREFIX           /data/data/com.termux/files/usr
ENV TMPDIR           /data/data/com.termux/files/usr/tmp
ENV TZ               UTC

COPY --from=bootstrap / /

WORKDIR /data/data/com.termux/files/home
SHELL ["/data/data/com.termux/files/usr/bin/sh", "-c"]
USER 1000:1000

CMD ["/data/data/com.termux/files/usr/bin/login"]

[sylirre]

Can't reproduce the issue:

~ $ echo -e "#include <iostream>\nint main() { std::cout << std::endl; }" > main.cpp
~ $ clang++ main.cpp
~ $ clang++  -std=c++11 main.cpp -v
clang version 15.0.7
Target: aarch64-unknown-linux-android24
Thread model: posix
InstalledDir: /data/data/com.termux/files/usr/bin
 "/data/data/com.termux/files/usr/bin/clang-15" -cc1 -triple aarch64-unknown-linux-android24 -emit-obj -mrelax-all --mrelax-relocations -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name main.cpp -mrelocation-model pic -pic-level 2 -pic-is-pie -mframe-pointer=non-leaf -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu generic -target-feature +neon -target-feature +v8a -target-feature +fix-cortex-a53-835769 -target-abi aapcs -fallow-half-arguments-and-returns -mllvm -treat-scalable-fixed-error-as-warning -debugger-tuning=gdb -v -fcoverage-compilation-dir=/data/data/com.termux/files/home -resource-dir /data/data/com.termux/files/usr/lib/clang/15.0.7 -isysroot /data/data/com.termux/files -internal-isystem /data/data/com.termux/files/usr/include/c++/v1 -internal-isystem /data/data/com.termux/files/usr/lib/clang/15.0.7/include -internal-isystem /data/data/com.termux/files/usr/local/include -internal-externc-isystem /data/data/com.termux/files/usr/include/aarch64-linux-android -internal-externc-isystem /data/data/com.termux/files/include -internal-externc-isystem /data/data/com.termux/files/usr/include -std=c++11 -fdeprecated-macro -fdebug-compilation-dir=/data/data/com.termux/files/home -ferror-limit 19 -fno-signed-char -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -target-feature +outline-atomics -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /data/data/com.termux/files/usr/tmp/main-5a2b15.o -x c++ main.cpp
clang -cc1 version 15.0.7 based upon LLVM 15.0.7 default target aarch64-unknown-linux-android24
ignoring nonexistent directory "/data/data/com.termux/files/usr/local/include"
ignoring nonexistent directory "/data/data/com.termux/files/include"
#include "..." search starts here:
#include <...> search starts here:
 /data/data/com.termux/files/usr/include/c++/v1
 /data/data/com.termux/files/usr/lib/clang/15.0.7/include
 /data/data/com.termux/files/usr/include/aarch64-linux-android
 /data/data/com.termux/files/usr/include
End of search list.
 "/data/data/com.termux/files/usr/bin/ld.lld" --sysroot=/data/data/com.termux/files -pie -EL --fix-cortex-a53-843419 -z now -z relro -z max-page-size=4096 --hash-style=gnu --enable-new-dtags -rpath=/data/data/com.termux/files/usr/lib --eh-frame-hdr -m aarch64linux -dynamic-linker /system/bin/linker64 -o a.out /data/data/com.termux/files/usr/lib/crtbegin_dynamic.o -L/data/data/com.termux/files/usr/lib -L/system/lib64 /data/data/com.termux/files/usr/tmp/main-5a2b15.o -lc++_shared /data/data/com.termux/files/usr/lib/clang/15.0.7/lib/linux/libclang_rt.builtins-aarch64-android.a -l:libunwind.a -ldl -lm -lc /data/data/com.termux/files/usr/lib/clang/15.0.7/lib/linux/libclang_rt.builtins-aarch64-android.a -l:libunwind.a -ldl /data/data/com.termux/files/usr/lib/crtend_android.o
~ $

QEMU initialized using

docker run --rm --privileged aptman/qus -s -- -p aarch64 arm

aptman/qus image hash: ca153c98a4b3.

Container started via this command:

docker run -it --security-opt seccomp:unconfined termux/termux-docker:aarch64

[licy183]

Emmm... It seems that this issue is unable to reproduce for me too using the latest aptman/qus. Maybe the qemu version matters. I'll close this issue. Maybe we should add a notice to README.md later.