Directory not writeable

[junghans]

For the full story see: https://bugzilla.redhat.com/show_bug.cgi?id=1897839

In short, a Dockerfile like:

FROM registry.fedoraproject.org/fedora:rawhide
RUN dnf -y update
RUN dnf -y install texlive
RUN ls -l /usr/share/texlive/
RUN whoami
RUN test -w /usr/share/texlive/texmf-config/

will fail, even though /usr/share/texlive/texmf-config/ is owned by root.

See https://github.com/junghans/rawhide-latex/runs/1439529976?check_suite_focus=true for example using docker/[email protected], but I can reproduce this locally with Docker version 19.03.13, build 4484c46 (Fedora) and Docker version 19.03.13, build 4484c46d9d (OSX).

The funny part is that podman and docker with bash works as expected, but the default RUN with sh fails (see https://bugzilla.redhat.com/show_bug.cgi?id=1897839#c18).

[thaJeztah]

Hmm.. yes, I'm able to reproduce;

DOCKER_BUILDKIT=0 docker build -t foo -<<EOF
FROM registry.fedoraproject.org/fedora:rawhide
RUN dnf -y update
RUN dnf -y install texlive
RUN ls -l /usr/share/texlive/
RUN whoami
RUN test -w /usr/share/texlive/texmf-config/
EOF

Sending build context to Docker daemon  2.048kB
Step 1/6 : FROM registry.fedoraproject.org/fedora:rawhide
 ---> bbbdb33fe265
Step 2/6 : RUN dnf -y update
 ---> Using cache
 ---> 75fcdf5ab120
Step 3/6 : RUN dnf -y install texlive
 ---> Using cache
 ---> ae9a069847be
Step 4/6 : RUN ls -l /usr/share/texlive/
 ---> Using cache
 ---> 4b48054739ed
Step 5/6 : RUN whoami
 ---> Using cache
 ---> 66a18e2ace18
Step 6/6 : RUN test -w /usr/share/texlive/texmf-config/
 ---> Running in 4d69d51e2ef9
The command '/bin/sh -c test -w /usr/share/texlive/texmf-config/' returned a non-zero code: 1

Trying to run that failed container, the test -w fails indeed;

docker run --rm 66a18e2ace18 sh -c 'test -w /usr/share/texlive/texmf-config/; echo $?'
1

Interesting bit is that it's just the test -w failing, because trying to write a file to that directory works;

docker run --rm 66a18e2ace18 sh -c 'touch /usr/share/texlive/texmf-config/somefile && echo > /usr/share/texlive/texmf-config/some-other-file && ls -la /usr/share/texlive/texmf-config/'

total 20
drwxr-xr-x 1 root root 4096 Nov 23 09:55 .
drwxr-xr-x 1 root root 4096 Nov 23 09:27 ..
-rw-r--r-- 1 root root    1 Nov 23 09:55 some-other-file
-rw-r--r-- 1 root root    0 Nov 23 09:55 somefile
drwxr-xr-x 2 root root 4096 Nov 23 09:27 web2c

The problem seems to be seccomp blocking a syscall; doing a docker run with seccomp disabled (--security-opt seccomp=unconfined) makes it work;

docker run --rm --security-opt seccomp=unconfined 66a18e2ace18 sh -c 'test -w /usr/share/texlive/texmf-config/; echo $?'
0

The funny part is that podman and docker with bash works as expected, but the default RUN with sh fails

bash or sh doesn't seem to make a difference, but is it possible that podman does not use seccomp by default?

So question is; what syscall is test -w attempting to make that's being blocked by seccomp? Possibly the test binary in the rawhide image uses a syscall that's either blocked by the profile or unknown in libseccomp on the host. Without seccomp blocking the syscall, binaries could fall back to using an alternative syscall (if the kernel reports the syscall as not being supported (ENOSYS)), but the seccomp profile as a default will return a EPERM (see a related discussion in opencontainers/runc#2151)

[thaJeztah]

@justincormack PTAL

[justincormack]

This is really strange, as the command fails, but if I strace it in the same build, then it succeeds.

execve("/usr/bin/test", ["test", "-w", "/usr/share/texlive/texmf-config/"], 0x7fff33df2e10 /* 10 vars */) = 0
brk(NULL)                               = 0x5608d4edc000
arch_prctl(0x3001 /* ARCH_??? */, 0x7ffc5f8abc20) = -1 EINVAL (Invalid argument)
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=20051, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 20051, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7ff90df60000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340|\2\0\0\0\0\0"..., 832) = 832
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
pread64(3, "\4\0\0\0\20\0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0", 32, 848) = 32
pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\"\367\0302\245>\310\25l\372/\3607\6Z\311"..., 68, 880) = 68
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=3230224, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff90df5e000
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
mmap(NULL, 1876624, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7ff90dd93000
mprotect(0x7ff90ddb9000, 1683456, PROT_NONE) = 0
mmap(0x7ff90ddb9000, 1372160, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26000) = 0x7ff90ddb9000
mmap(0x7ff90df08000, 307200, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x175000) = 0x7ff90df08000
mmap(0x7ff90df54000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c0000) = 0x7ff90df54000
mmap(0x7ff90df5a000, 12944, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7ff90df5a000
close(3)                                = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff90dd91000
arch_prctl(ARCH_SET_FS, 0x7ff90df5f600) = 0
mprotect(0x7ff90df54000, 12288, PROT_READ) = 0
mprotect(0x5608d44ed000, 4096, PROT_READ) = 0
mprotect(0x7ff90df92000, 4096, PROT_READ) = 0
munmap(0x7ff90df60000, 20051)           = 0
brk(NULL)                               = 0x5608d4edc000
brk(0x5608d4efd000)                     = 0x5608d4efd000
brk(NULL)                               = 0x5608d4efd000
newfstatat(AT_FDCWD, "/usr/share/texlive/texmf-config/", {st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0
geteuid()                               = 0
getegid()                               = 0
getuid()                                = 0
getgid()                                = 0
access("/usr/share/texlive/texmf-config/", W_OK) = 0
close(1)                                = 0
close(2)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++

[justincormack]

Ah interesting, if I run it with seccomp disabled, it behaves differently, loading locale files. That is really strange, something different must have happened earlier as it just makes a decision not to do this, without anything failing.

[junghans]

The directory is created by installing texlive.

Interestingly on the previous version of Fedora, F33, registry.fedoraproject.org/fedora:33 it works fine as well.

/cc @cverna

[cverna]

I think this is the reason behind this issue --> faccessat() is broken when running under Docker

So pretty much the a bug in glibc in Fedora rawhide

[junghans]

Here is another, even simpler, case that fails for the same reason:

FROM registry.fedoraproject.org/fedora:rawhide
RUN echo "echo test" > test.sh
RUN chmod +x test.sh
RUN ls -l test.sh
RUN test -x test.sh

(log e.g. here: https://github.com/junghans/rawhide-latex/pull/2/checks?check_run_id=1480967204)

[tomalok]

I think this may be related to https://gitlab.alpinelinux.org/alpine/aports/-/issues/12321 -- musl-1.2.2 (part of alpine edge) now supports the faccessat2 system call. Skimming through the moby Dockerfile, it seems that Docker's build is using Debian Buster, which would pull in libseccomp-2.3.3-4, but at least libseccomp-2.4.4 is needed...

[thaJeztah]

it seems that Docker's build is using Debian Buster

The .deb and .rpm packages are built using a base-image that's matching the target platform, e.g.; https://github.com/docker/docker-ce-packaging/tree/100f31916b938002df9d78c630ce2ed7ea2e4fa5/rpm/fedora-32

[hberntsen]

This is really strange, as the command fails, but if I strace it in the same build, then it succeeds.

execve("/usr/bin/test", ["test", "-w", "/usr/share/texlive/texmf-config/"], 0x7fff33df2e10 /* 10 vars */) = 0
brk(NULL)                               = 0x5608d4edc000
arch_prctl(0x3001 /* ARCH_??? */, 0x7ffc5f8abc20) = -1 EINVAL (Invalid argument)
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=20051, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 20051, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7ff90df60000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340|\2\0\0\0\0\0"..., 832) = 832
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
pread64(3, "\4\0\0\0\20\0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0", 32, 848) = 32
pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\"\367\0302\245>\310\25l\372/\3607\6Z\311"..., 68, 880) = 68
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=3230224, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff90df5e000
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
mmap(NULL, 1876624, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7ff90dd93000
mprotect(0x7ff90ddb9000, 1683456, PROT_NONE) = 0
mmap(0x7ff90ddb9000, 1372160, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26000) = 0x7ff90ddb9000
mmap(0x7ff90df08000, 307200, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x175000) = 0x7ff90df08000
mmap(0x7ff90df54000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c0000) = 0x7ff90df54000
mmap(0x7ff90df5a000, 12944, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7ff90df5a000
close(3)                                = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff90dd91000
arch_prctl(ARCH_SET_FS, 0x7ff90df5f600) = 0
mprotect(0x7ff90df54000, 12288, PROT_READ) = 0
mprotect(0x5608d44ed000, 4096, PROT_READ) = 0
mprotect(0x7ff90df92000, 4096, PROT_READ) = 0
munmap(0x7ff90df60000, 20051)           = 0
brk(NULL)                               = 0x5608d4edc000
brk(0x5608d4efd000)                     = 0x5608d4efd000
brk(NULL)                               = 0x5608d4efd000
newfstatat(AT_FDCWD, "/usr/share/texlive/texmf-config/", {st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0
geteuid()                               = 0
getegid()                               = 0
getuid()                                = 0
getgid()                                = 0
access("/usr/share/texlive/texmf-config/", W_OK) = 0
close(1)                                = 0
close(2)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++

Just went down the same rabbit hole on Debian 10. Note that the test function in bash is a builtin function and you are testing /usr/bin/test. That's why strace worked