Mixed-execution mode gets in the way

[dleonard0]

Expected Behavior

proot -q qemu-x86_64 -r romfs bin/sh -c true

Actual Behavior

proot -q qemu-x86_64 -r romfs bin/sh -c true
proot info: '/.../romfs/bin/sh' is a host ELF
proot error: execve("/bin/sh"): No such file or directory
proot info: possible causes:
  * the program is a script but its interpreter (eg. /bin/sh) was not found;
  * the program is an ELF but its interpreter (eg. ld-linux.so) was not found;
  * the program is a foreign binary but qemu was not specified;
  * qemu does not work correctly (if specified);
  * the loader was not found or doesn't work.
fatal error: see `proot --help`.

The issue (I think) is that romfs/bin/sh appears to proot to be a native ELF executable, so it enters mixed-execution mode using preserved native LD_LIBRARY_PATH. The trouble is that my romfs/bin/sh was built using a different ld.so and libc (musl), so the elf interp won't be found. I'd also like to run it under the (potentially modified) QEMU that i've specified.

I'm working on adding a --no-host-elf flag to see if that helps.

Specifications

• Proot/Care version: 5.1.0
• Kernel version: 5.15.0-57-generic
• Host distribution: Ubuntu 22.04.1 LTS
• Guest distribution: custom (musl, busybox)

Log

(replaced workdir with "D")

+ proot '-i' '0:0' '-r' 'bin/../romfs' '-w' '/mnt' '-b' '/tmp/tmp.L3O0rU1D5N:/var' '-b' '.:/mnt' '-b' '/tmp/tmp.xc3U1Nnf5P:/etc/config' '-b' '/proc' '-b' '/dev' '-b' '/tmp/tmp.nxVmoEfBXB:/proc/mounts!' '-q' 'qemu-x86_64 -r 6.0.0' '-v' '3' '/bin/sh' '-c' 'true'
proot info: binding = /dev/null:/etc/ld.so.preload
proot info: binding = /:/host-rootfs
proot info: binding = /tmp/tmp.nxVmoEfBXB:/proc/mounts
proot info: binding = /dev
proot info: binding = /proc
proot info: binding = /tmp/tmp.xc3U1Nnf5P:/etc/config
proot info: binding = D:/mnt
proot info: binding = /tmp/tmp.L3O0rU1D5N:/var
proot info: binding = D/romfs:/
proot info: vpid 1: translate("/" + "/bin/sh")
proot info: vpid 1:          -> "D/romfs/bin/sh"
proot info: vpid 1: translate("/" + "/bin/sh")
proot info: vpid 1:          -> "D/romfs/bin/sh"
proot info: host rootfs = /host-rootfs
proot info: glue rootfs = /tmp/proot-1751091-zeBnj5
proot info: exe = /bin/sh
proot info: argv = /bin/sh -c true
proot info: qemu = /usr/bin/qemu-x86_64 -r 6.0.0
proot info: initial cwd = /mnt
proot info: verbose level = 3
proot info: pid 1751091: access to "/dev/pts/4" (fd 0) won't be translated until closed
proot info: pid 1751091: access to "/dev/pts/4" (fd 1) won't be translated until closed
proot info: pid 1751091: access to "/dev/pts/4" (fd 2) won't be translated until closed
proot info: pid 1751091: access to "/proc/1751091/fd" (fd 3) won't be translated until closed
proot info: ptrace acceleration (seccomp mode 2) enabled
proot info: vpid 1: sysenter start: prctl(0x26, 0x1, 0x0, 0x0, 0x0, 0x5646a7d3e2b0) = 0xffffffffffffffda [0x7fff28cdb620, 0]
proot info: vpid 1: sysenter start: execve(0x5646a7d2bc70, 0x7fff28cdb9b0, 0x7fff28cdb9d0, 0x7facf935f531, 0x0, 0x5646a7d3e2b0) = 0xffffffffffffffda [0x7fff28cdb5e8, 0]
proot info: vpid 1: translate("/" + "/bin/sh")
proot info: vpid 1:          -> "D/romfs/bin/sh"
proot info: 'D/romfs/bin/sh' is a host ELF
proot info: vpid 1: translate("/" + "/host-rootfs/lib/ld-musl-x86_64.so.1")
proot info: vpid 1:          -> "/usr/lib/ld-musl-x86_64.so.1"
proot error: execve("/bin/sh"): No such file or directory
proot info: possible causes:
  * the program is a script but its interpreter (eg. /bin/sh) was not found;
  * the program is an ELF but its interpreter (eg. ld-linux.so) was not found;
  * the program is a foreign binary but qemu was not specified;
  * qemu does not work correctly (if specified);
  * the loader was not found or doesn't work.
fatal error: see `proot --help`.
proot info: vpid 1: sysenter start: newfstatat(0xffffff9c, 0x5646a7d3d890, 0x7fff28cdb4b0, 0x100, 0x21, 0x7fffffff) = 0xffffffffffffffda [0x7fff28cdb4a8, 0]
proot info: vpid 1: translate("/" + "D/romfs/etc/ld.so.preload")
proot info: vpid 1: sysenter start: newfstatat(0xffffff9c, 0x5646a7d3d6d0, 0x7fff28cdb4b0, 0x100, 0x5646a7d3d830, 0x7fffffff) = 0xffffffffffffffda [0x7fff28cdb4a8, 0]
proot info: vpid 1: translate("/" + "D/romfs/host-rootfs")
proot info: vpid 1: sysenter start: newfstatat(0xffffff9c, 0x7facf9413072, 0x7fff28cdb400, 0x0, 0x5646a7d3d670, 0x7fffffff) = 0xffffffffffffffda [0x7fff28cdb3f8, 0]
proot info: vpid 1: translate("/mnt" + ".")
proot info: vpid 1:          -> "D/."
proot info: vpid 1: sysenter start: newfstatat(0xffffff9c, 0x5646a7d2de64, 0x7fff28cdb490, 0x0, 0x5646a7d3d670, 0x7fffffff) = 0xffffffffffffffda [0x7fff28cdb3f8, 0]
proot info: vpid 1: translate("/" + "/mnt")
proot info: vpid 1:          -> "D"
proot info: vpid 1: sysenter start: chmod(0x5646a7d3d640, 0x1c0, 0x0, 0x0, 0x5646a7d29010, 0x5646a7d2bc80) = 0xffffffffffffffda [0x7fff28cdb538, 0]
proot info: vpid 1: translate("/" + "/tmp/proot-1751091-zeBnj5")
proot info: vpid 1:          -> "D/romfs/tmp/proot-1751091-zeBnj5"
proot error: can't chmod '/tmp/proot-1751091-zeBnj5': No such file or directory
proot info: vpid 1: sysenter start: chdir(0x5646a7d2bc80, 0x7fff28cd8e80, 0x0, 0x0, 0x1a, 0x7fffffff) = 0xffffffffffffffda [0x7fff28cdb538, 0]
proot info: vpid 1: translate("/" + "/mnt/.")
proot info: vpid 1:          -> "D/."
proot info: vpid 1: exited with status 1

[oxr463]

Wow, that’s very interesting. I’m curious about how the proposed flag will work. Is this only for environments with glibc and musl on the same system?

[dleonard0]

Wow, that’s very interesting. I’m curious about how the proposed flag will work.

My attempt currently uses an option to disable a single block in expand_runner(); see master...dleonard0:proot:no-host-elf

Is this only for environments with glibc and musl on the same system?

That would be my case, but it is more general than that. My case is building a sysroot for an embedded target and wanting to run some small unit tests during build. I suppose what I have is a case where proot's current mixed execution assumption isn't right for me, though I can understand why it is the default.

Update: fixed logic bug, in my attempt; updated URL above

[dleonard0]

My only regret here here my use of the double-negative form !no_host_elf and that host-elf is not very user-friendly ... I would be happy to receive direction to improve that... maybe naming it --no-mixed-mode or something?

[oxr463]

My only regret here here my use of the double-negative form !no_host_elf and that host-elf is not very user-friendly ... I would be happy to receive direction to improve that... maybe naming it --no-mixed-mode or something?

How about --mix-mode=false? What if we make it a boolean?

[oxr463]

@dleonard0 please test the latest changes on my new branch. I changed the flag to accept a boolean value but it keeps crashing when I run it:

proot --mixed-mode true -v 1
proot info: binding = /
proot info: exe = /bin/sh
proot info: argv =
proot info: initial cwd = /usr/src/proot
proot info: verbose level = 1
proot info: pid 2491: access to "/dev/pts/0" (fd 0) won't be translated until closed
proot info: pid 2491: access to "/dev/pts/0" (fd 1) won't be translated until closed
proot info: pid 2491: access to "/dev/pts/0" (fd 2) won't be translated until closed
proot info: pid 2491: access to "/proc/2491/fd" (fd 3) won't be translated until closed
proot info: ptrace acceleration (seccomp mode 2) enabled
proot error: execve("/bin/sh"): Bad address
proot info: possible causes:
  * the program is a script but its interpreter (eg. /bin/sh) was not found;
  * the program is an ELF but its interpreter (eg. ld-linux.so) was not found;
  * the program is a foreign binary but qemu was not specified;
  * qemu does not work correctly (if specified);
  * the loader was not found or doesn't work.
fatal error: see `proot --help`.
proot info: vpid 1: exited with status 1

proot --mixed-mode false -v 1
proot info: binding = /
proot info: exe = /bin/sh
proot info: argv =
proot info: initial cwd = /usr/src/proot
proot info: verbose level = 1
proot info: pid 2493: access to "/dev/pts/0" (fd 0) won't be translated until closed
proot info: pid 2493: access to "/dev/pts/0" (fd 1) won't be translated until closed
proot info: pid 2493: access to "/dev/pts/0" (fd 2) won't be translated until closed
proot info: pid 2493: access to "/proc/2493/fd" (fd 3) won't be translated until closed
proot info: ptrace acceleration (seccomp mode 2) enabled
proot error: execve("/bin/sh"): Bad address
proot info: possible causes:
  * the program is a script but its interpreter (eg. /bin/sh) was not found;
  * the program is an ELF but its interpreter (eg. ld-linux.so) was not found;
  * the program is a foreign binary but qemu was not specified;
  * qemu does not work correctly (if specified);
  * the loader was not found or doesn't work.
fatal error: see `proot --help`.
proot info: vpid 1: exited with status 1