Failed to setup UID/GID map

[ghost]

I'm trying to run docker-in-docker (dind) (more specifically, docker:22.06.0-beta.0-dind-rootless) image on a very bare-bones Linux installation, but whenever i try to run the container, i get the following error message:

[rootlesskit:parent] error: failed to setup UID/GID map: newuidmap 493 [0 1000 1 1 100000 65536] failed: newuidmap: Could not set caps

The aforementioned Linux host (running on a VirtualBox VM) is composed solely of the following components:

• kernel 5.18.4
• busybox 1.35.0 (a reduced set of the utils it provides)
• iptables 1.8.8 (docker dependency)
• docker 20.10.9 (installed from a pre-built binary)
• shadow-subids 4.10 (provides newuidmap and newgidmap binaries, added on an attempt to fix the problem, not sure if that should be on the container side)

Everything listed above (with the exception of docker) is built from source, statically linked and customized to be as minimal as possible

[tianon]

Are you running with --privileged?

[ghost]

Yep, here's the command i'm using to run the container:

docker run --rm --pull=never --privileged --ulimit core=0 -it docker:22.06.0-beta.0-dind-rootless

[tianon]

Hmm, that should work -- maybe something similar to moby/moby#41812 ? Did your image get corrupted somehow or was unable to set the extended capabilities on the newuidmap after the image was extracted?

[ghost]

I'm not sure if that's relevant to the issue, but as the error message says something that i believe might be related to "capabilities" on "Could not set caps", i've tried running setuid on both newuidmap and newgidmap binaries, the result was the same below:

$ getcap newuidmap newgidmap
newuidmap cap_setuid=ep
newgidmap cap_setgid=ep

---

I think it's also worth mentioning that the configure script from shadow-subids source provides two flags: --with-fcaps and --without-fcaps which i also believe might have some correlation with the problem, but after trying to compile the binaries with each one of those flags, the same error persists

Below are the commands i use to compile shadow-subids from source:

./configure \
    --prefix="$PWD/build" \
    --disable-shared \
    --disable-account-tools-setuid \
    --disable-nls \
    --without-audit \
    --without-libpam \
    --without-selinux \
    --without-acl \
    --without-attr \
    --without-tcb \
    --without-yescrypt \
    --without-nscd \
    --without-group-name-max-length \
    --with-fcaps # or --without-fcaps
make LDFLAGS="--static -s" -j "$(nproc)"
make -j "$(nproc)" install

[yosifkit]

newuidmap and newgidmap don't need to exist in the host filesystem. They need to have the extended attributes within the container. If they do not, then the process of pulling and extracting the image failed somewhere (corruption or something else).

[ghost]

Due to the fact that i don't have an internet connection on this VM, I'm currently saving (and compressing) the dind container with the following command:

docker save docker:22.06.0-beta.0-dind-rootless | gzip > dind.tar.gz

I've extracted the compressed container archive on the VM, and tried inspecting the file capabilities of both newuidmap and newgidmap:

docker run --rm --privileged docker:22.06.0-beta.0-dind-rootless getcap /usr/bin/newuidmap /usr/bin/newgidmap

And i've got an empty output, i'm not sure if the compression has anything to do with the extracted image being corrupted, or if i should export and import it instead of save and load

[tianon]

Are either of the host you're doing docker save or your VM using AUFS (or a similar alternative graph driver that isn't overlay2/overlay)?

For reference, here's what it should look like:

$ docker run --rm docker:22.06-rc-dind-rootless getcap /usr/bin/newuidmap /usr/bin/newgidmap
/usr/bin/newuidmap cap_setuid=ep
/usr/bin/newgidmap cap_setgid=ep

[ghost]

I don't think so, from what i've saw on the docs:

overlay2 is the preferred storage driver for all currently supported Linux distributions, and requires no extra configuration.

The host where i'm running docker save is using ext4, and i haven't done any extra configuration when it comes to storage drivers, so i believe i'm using overlay2 which seems to be the default option

I'll try some others ideas i have in mind and share the results here as soon as i can

[ghost]

I've tried running an interactive session on the dind container (as root) to see if by manually setting the proper capabilites on newuidmap and newgidmap would make docker-entrypoint.sh stop complaining:

$ setcap cap_setuid+ep /usr/bin/newuidmap
Failed to set capabilities on file '/usr/bin/newuidmap': Not supported

After some further inspection, i found out that the setcap binary on the container doesn't have cap_setfcap, hence it can't add file capabilities to itself to other files (which is good, otherwise that would be a concerning security breach for a rootless user)

---

I even tried not gzipping the archive and repeating the process with both save/load and export/import to check if this could be causing any corruption, but it didn't worked either

It seems like for some reason, any file capabilities are instantly dropped once i try to run any docker image on the VM (it works fine on bare metal), and i also couldn't find any kernel config that seems to enable/disable support for file capabilites, so i assume that's enabled by default (at least for ext4)

[ghost]

@tianon @yosifkit i've figured out what was wrong, my kernel was lacking support for extended attributes, i've enabled it by compiling with the option CONFIG_EXT4_FS_SECURITY (Ext4 Security Labels) and now both newuidmap and newgidmap retain their proper file capabilities when running the container

Thanks a lot for your help on this!