Cross compile on aarch64 from MacOS/darwin to aarch64-unknown-linux-musl is failing with undefined symbol: readdir64

[reubenmiller]

When building with the newly release cargo-zigbuild version 0.17.2, the build fails with the following errors. From a quick inspection it looks to be related to #172 (though it is just a guess). Reverting back to cargo-zigbuild 0.17.1 resolves the issue.

The error that I get when cross compiling from MacOS M1 to aarch64-unknown-linux-musl is:

error: linking with `/Users/reubenmiller/Library/Caches/cargo-zigbuild/0.17.2/zigcc-aarch64-unknown-linux-musl.sh` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/Users/reubenmiller/.rustup/toolchains/1.70-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin:/Users/reubenmiller/.rustup/toolchains/1.70-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin/self-contained:/Users/reubenmiller/dev/projects/tedge/tmp/thin-edge.io/tests/RobotFramework/.venv/bin:/Users/reubenmiller/.nvm/versions/node/v16.20.0/bin:/Users/reubenmiller/homebrew/opt/gnu-tar/libexec/gnubin:/Users/reubenmiller/go/bin:/Users/reubenmiller/.gem/ruby/2.6.0/bin:/Users/reubenmiller/.local/bin:/Users/reubenmiller/homebrew/bin:/Users/reubenmiller/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Users/reubenmiller/.nvm/versions/node/v16.20.0/bin:/Users/reubenmiller/homebrew/opt/gnu-tar/libexec/gnubin:/Users/reubenmiller/go/bin:/Users/reubenmiller/.gem/ruby/2.6.0/bin:/Users/reubenmiller/.local/bin:/Users/reubenmiller/homebrew/bin:/Users/reubenmiller/homebrew/sbin:/Users/reubenmiller/.cargo/bin" VSLANG="1033" "/Users/reubenmiller/Library/Caches/cargo-zigbuild/0.17.2/zigcc-aarch64-unknown-linux-musl.sh" "/Users/reubenmiller/.rustup/toolchains/1.70-aarch64-apple-darwin/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crt1.o" "/Users/reubenmiller/.rustup/toolchains/1.70-aarch64-apple-darwin/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crti.o" "/Users/reubenmiller/.rustup/toolchains/1.70-aarch64-apple-darwin/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crtbegin.o" "/var/folders/wn/06nj6dqj07db9tbsh_094wnr0000gn/T/rustckaNAp4/symbols.o" "/Users/reubenmiller/dev/projects/tedge/tmp/thin-edge.io/target/aarch64-unknown-linux-musl/release/deps/sawtooth_publisher-8617a97078f8b92f.sawtooth_publisher.9260d1d0-cgu.0.rcgu.o" "-Wl,--as-needed" "-L" "/Users/reubenmiller/dev/projects/tedge/tmp/thin-edge.io/target/aarch64-unknown-linux-musl/release/deps" "-L" "/Users/reubenmiller/dev/projects/tedge/tmp/thin-edge.io/target/release/deps" "-L" "/Users/reubenmiller/dev/projects/tedge/tmp/thin-edge.io/target/aarch64-unknown-linux-musl/release/build/ring-0931aa783d916770/out" "-L" "/Users/reubenmiller/.rustup/toolchains/1.70-aarch64-apple-darwin/lib/rustlib/aarch64-unknown-linux-musl/lib" "-Wl,-Bstatic" "/var/folders/wn/06nj6dqj07db9tbsh_094wnr0000gn/T/rustckaNAp4/libring-751c867e148decbc.rlib" "-lunwind" "-lc" "/Users/reubenmiller/.rustup/toolchains/1.70-aarch64-apple-darwin/lib/rustlib/aarch64-unknown-linux-musl/lib/libcompiler_builtins-a546a2beee625b76.rlib" "-Wl,-Bdynamic" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-nostartfiles" "-L" "/Users/reubenmiller/.rustup/toolchains/1.70-aarch64-apple-darwin/lib/rustlib/aarch64-unknown-linux-musl/lib" "-L" "/Users/reubenmiller/.rustup/toolchains/1.70-aarch64-apple-darwin/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained" "-o" "/Users/reubenmiller/dev/projects/tedge/tmp/thin-edge.io/target/aarch64-unknown-linux-musl/release/deps/sawtooth_publisher-8617a97078f8b92f" "-Wl,--gc-sections" "-static" "-no-pie" "-Wl,-z,relro,-z,now" "-Wl,--strip-all" "-nodefaultlibs" "/Users/reubenmiller/.rustup/toolchains/1.70-aarch64-apple-darwin/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crtend.o" "/Users/reubenmiller/.rustup/toolchains/1.70-aarch64-apple-darwin/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crtn.o"
  = note: ld.lld: error: undefined symbol: readdir64
          >>> referenced by sawtooth_publisher.9260d1d0-cgu.0
          >>>               /Users/reubenmiller/dev/projects/tedge/tmp/thin-edge.io/target/aarch64-unknown-linux-musl/release/deps/sawtooth_publisher-8617a97078f8b92f.sawtooth_publisher.9260d1d0-cgu.0.rcgu.o:(OUTLINED_FUNCTION_5589)

Version information

$ rustc --version
rustc 1.70.0 (90c541806 2023-05-31

$ cargo --version
cargo 1.70.0 (ec8a8a0ca 2023-04-25)

$ python3 -m ziglang version
0.11.0

$ cargo-zigbuild --version
cargo-zigbuild 0.17.1

[messense]

cc @haohaolee Looks like the weak symbol list isn't complete?

https://github.com/search?q=repo%3Abminor%2Fmusl%20_LARGEFILE64_SOURCE&type=code

[messense]

@reubenmiller IMO it should be easy to fix by adding more LFS64 functions to

cargo-zigbuild/src/linux/mod.rs

Lines 22 to 27 in 846edea

	pub static MUSL_WEAK_SYMBOLS_MAPPING_SCRIPT: &str = r#"
	PROVIDE (open64 = open);
	PROVIDE (stat64 = stat);
	PROVIDE (fstat64 = fstat);
	PROVIDE (lseek64 = lseek);
	"#;

Happy to accept a PR to do that.

[reubenmiller]

I can confirm that the build experienced will be fixed once #180 is merged.

[haohaolee]

I wanted to be conservative so I tried to modify as few symbols as possible in #172, I was afraid that too many adds lead to other issues.

Hi @reubenmiller does your project rely on libc crate? as far as I know, some symbols like readdir64 are referenced in libc crate, and the latest libc crate has already fixed this issue.

Please let me know if this is the case, thanks

[haohaolee]

I have managed to figure out this, I did miss some symbols in my PR #172:

nm ~/.rustup/toolchains/1.71.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-493936ddc336b3cf.rlib | grep "64$"
nm: lib.rmeta: no symbols
                 U fstat64
                 U fstatat64
                 U ftruncate64
  ......
                 U lseek64
                 U lstat64
                 U open64
                 U pread64
                 U pwrite64
                 U readdir64
                 U stat64

At least these are the symbols needed.
And for 1.72.0, aka stable:

nm ~/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-aa8203b3ba116fbe.rlib | grep "64$"
nm: lib.rmeta: no symbols
0000000000000000 r GCC_except_table164
   ......

It would be better to have a test for this regression.

[reubenmiller]

I wanted to be conservative so I tried to modify as few symbols as possible in #172, I was afraid that too many adds lead to other issues.

Hi @reubenmiller does your project rely on libc crate? as far as I know, some symbols like readdir64 are referenced in libc crate, and the latest libc crate has already fixed this issue.

Please let me know if this is the case, thanks

No we don't use anything from the libc crate, we use musl builds for portability reasons.

[haohaolee]

@reubenmiller No worries. Ignore me, I was asking a wrong question. I have checked your build and I think I have a better understanding now, this issue has nothing to do with the libc crate.

In short, you need to satisfy those undefined symbols in std, e.g. U readdir64

ps: musl is a kinda libc, and your project does depend on libc crate if you check the Cargo.lock