MAMBO fails to build on Arch Linux

[IgWod]

It looks like MAMBO fails to build on the recent versions of Arch Linux. The problem has been initially reported by @ijan1.

The compilation error can be seen below:

MAMBO: detected architecture "aarch64"
PIE: detected architecture "aarch64"
make --no-print-directory ARCH=a64 pie
ruby generate_decoder.rb a64  > pie-a64-decoder.c
ruby generate_decoder.rb a64 header  > pie-a64-decoder.h
cc -c -Os -Wall -g -std=c99  pie-a64-decoder.c -o pie-a64-decoder.o
ruby generate_encoder.rb a64  > pie-a64-encoder.c
ruby generate_encoder.rb a64 header  > pie-a64-encoder.h
cc -c -Os -Wall -g -std=c99  pie-a64-encoder.c -o pie-a64-encoder.o
ruby generate_field-decoder.rb a64  > pie-a64-field-decoder.c
ruby generate_field-decoder.rb a64 header  > pie-a64-field-decoder.h
cc -c -Os -Wall -g -std=c99  pie-a64-field-decoder.c -o pie-a64-field-decoder.o
ruby api/generate_emit_wrapper.rb pie/pie-a64-encoder.c header > api/emit_a64.h
cc -D_GNU_SOURCE -g -std=gnu99 -O2 -Wunused-variable -DVERSION=\"583f3b7e-dirty\" -c -o elf/elf_loader.o elf/elf_loader.c
cc -D_GNU_SOURCE -g -std=gnu99 -O2 -Wunused-variable -DVERSION=\"583f3b7e-dirty\"   -c -o elf/symbol_parser.o elf/symbol_parser.c
ruby api/generate_emit_wrapper.rb pie/pie-a64-encoder.c > api/emit_a64.c
cc -D_GNU_SOURCE -g -std=gnu99 -O2 -Wunused-variable -DVERSION=\"583f3b7e-dirty\" -static -ldl -Wl,-Ttext-segment=0x7000000000 -DDBM_LINK_UNCOND_IMM -DDBM_INLINE_UNCOND_IMM -DDBM_LINK_COND_IMM -DDBM_LINK_CBZ -DDBM_LINK_TBZ -DDBM_TB_DIRECT  -DLINK_BX_ALT -DDBM_INLINE_HASH -DDBM_TRACES  -I/usr/include/libelf -I. -o dbm common.c dbm.c traces.c syscalls.c dispatcher.c signals.c util.S api/helpers.c api/plugin_support.c api/branch_decoder_support.c api/load_store.c api/internal.c api/hash_table.c elf/elf_loader.o elf/symbol_parser.o arch/aarch64/dispatcher_aarch64.S arch/aarch64/dispatcher_aarch64.c arch/aarch64/scanner_a64.c api/emit_a64.c  pie/pie-a64-field-decoder.o pie/pie-a64-encoder.o pie/pie-a64-decoder.o -lelf -lz -pthread 
/sbin/ld: /usr/lib/gcc/aarch64-unknown-linux-gnu/12.1.0/../../../../lib/libelf.a(elf_compress.o): in function `__libelf_compress':
(.text+0xe8): undefined reference to `ZSTD_createCCtx'
/sbin/ld: (.text+0xec): undefined reference to `ZSTD_createCCtx'
/sbin/ld: (.text+0xf4): undefined reference to `ZSTD_isError'
/sbin/ld: (.text+0xf8): undefined reference to `ZSTD_isError'
/sbin/ld: (.text+0x118): undefined reference to `ZSTD_compressStream2'
/sbin/ld: (.text+0x11c): undefined reference to `ZSTD_compressStream2'
/sbin/ld: (.text+0x424): undefined reference to `ZSTD_freeCCtx'
/sbin/ld: (.text+0x428): undefined reference to `ZSTD_freeCCtx'
/sbin/ld: (.text+0x4f4): undefined reference to `ZSTD_freeCCtx'
/sbin/ld: (.text+0x4f8): undefined reference to `ZSTD_freeCCtx'
/sbin/ld: (.text+0x520): undefined reference to `ZSTD_freeCCtx'
/sbin/ld: /usr/lib/gcc/aarch64-unknown-linux-gnu/12.1.0/../../../../lib/libelf.a(elf_compress.o):(.text+0x524): more undefined references to `ZSTD_freeCCtx' follow
/sbin/ld: /usr/lib/gcc/aarch64-unknown-linux-gnu/12.1.0/../../../../lib/libelf.a(elf_compress.o): in function `__libelf_decompress':
(.text+0x9c8): undefined reference to `ZSTD_decompress'
/sbin/ld: (.text+0x9cc): undefined reference to `ZSTD_decompress'
/sbin/ld: (.text+0x9e0): undefined reference to `ZSTD_isError'
/sbin/ld: (.text+0x9e4): undefined reference to `ZSTD_isError'
collect2: error: ld returned 1 exit status
make[1]: *** [makefile:73: dbm] Error 1
make: *** [makefile:64: all] Error 2

To fix this problem, we first need to add libzstd to the makefile:

-LIBS=-lelf -lpthread -lz
+LIBS=-lelf -lz -lzstd -pthread

However, this still fails, as static libzstd cannot be found. The root cause of it is that Arch Linux removed the static version of libzstd from its packages: https://bugs.archlinux.org/task/78526.

To get it to work, libzstd has to be built locally (https://github.com/facebook/zstd), and libzstd.a installed in /usr/lib. This enables MAMBO to build and run.

Since MAMBO relies on libelf, that relies on libz and libzstd, removing the dependency is not simple.

We need to discuss it, how to handle it properly. Any suggestions are welcome! (cc @jkressel @GuillermoCallaghan)

EDIT:

Tested in chroot on Zenfone 9 with following Arch Linux configuration:

NAME="Arch Linux ARM"
PRETTY_NAME="Arch Linux ARM"
ID=archarm
ID_LIKE=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinuxarm.org/"
DOCUMENTATION_URL="https://archlinuxarm.org/wiki"
SUPPORT_URL="https://archlinuxarm.org/forum"
BUG_REPORT_URL="https://github.com/archlinuxarm/PKGBUILDs/issues"
LOGO=archlinux-logo

[ijan1]

Hi Igor,

I don't think there's a clean way to handle this other than having cases for handling disto-specific package quirks, like not providing a static library.
Although, from reading around it does seems like that dynamically linked libraries are preferred by package managers.[1]
While I couldn't find something more authorative, the whole Arch Linux page for Haskell does seem to mention that the packages provided by Arch are dynamically linked, whereas GHC is set-up for static compilation by default which seems to lead to a lot of frustration for Haskell devs.[2][3]

Debian (and Ubuntu by extension) seems to be leaning towards the same idea of providing dynamically linked libraries first, and then providing the static ones in a separate -dev package together with the header files.[4] So Debian-based distros are at least guaranteed to have a statically compiled version of the libraries we are using.

As for the more practical concerns; having Mambo be dynamically compiled by default is fine, if there's no big performance regressions. I don't know if there's any plans of providing precompiled binaries, in which case being able to statically compile to avoid DLL hell would be nice for distribution purposes.
In the case of a performance regression, a possibility would be to provide the dependencies ourselves, like PIE.

[1] https://old.reddit.com/r/haskell/comments/7jyie0/the_arch_linux_community_does_not_look_very_about/draeoji/
[2] https://wiki.archlinux.org/title/haskell#Configuration
[3] https://old.reddit.com/r/archlinux/comments/gutkbu/setting_up_haskell_devel_environment_on_arch/
[4] https://www.debian.org/doc/debian-policy/ch-sharedlibs.html#static-libraries