Mounted volume changes user ID's of files

[gigabit942007]

/kind bug
Volumes mounted inside of containers change ownership of actual files.

Steps to reproduce the issue:

1. Create a container that takes ownership of files

2. Add to the container external storage with -v

3. Files within the volume mounted via -v will have the owner from inside the container.

Describe the results you received:
Docker would keep the user ID's consistent:

drwxr-xr-x 2 pingu pingu 4096 Jul 15 12:04 DockerFolder
drwxrwxrwx 2 100910 100910 4096 Jul 22 20:16 PodmanFolder

Describe the results you expected:
I would expect that at least the ID's of files are kept.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      2.0.2
API Version:  1
Go Version:   go1.14.3
Built:        Thu Jan  1 02:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

`host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.18-1.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.18, commit: 6e8799f576f11f902cd8a8d8b45b2b2caf636a85'
  cpus: 2
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: file
  hostname: pingu
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.7.9-200.fc32.x86_64
  linkmode: dynamic
  memFree: 70103040
  memTotal: 1677991936
  ociRuntime:
    name: crun
    package: crun-0.14.1-1.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.14.1
      commit: 598ea5e192ca12d4f6378217d3ab1415efeddefa
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 3652587520
  swapTotal: 3665817600
  uptime: 6h 46m 51.68s (Approximately 0.25 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/pingu/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 1
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.2-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/pingu/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 4
  runRoot: /run/user/1000/containers
  volumePath: /home/pingu/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Thu Jan  1 02:00:00 1970
  GitCommit: ""
  GoVersion: go1.14.3
  OsArch: linux/amd64
  Version: 2.0.2
`

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.0.2-1.fc32.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):
Running on baremetal Fedora 32.

[rhatdan]

Could you give us a simple reproducer of the command, is this because you are comparing rootless versus rootful.
If you run podman as root, does it work the same as Docker?
In rootless mode the UID in the homedir will not be the same as the UID within the container, since it uses user namespace.
If you do a podman unshare and look at the files does it have the expected uid?

podman unshare ls -l PodmanFolder

[gigabit942007]

For some reason some of the description is missing from what i wrote. Additional details bellow:

I have made the following steps:
Created this script:

podman create \
  --name=syncthing \
  --userns=keep-id \
  -e STGUIADDRESS=0.0.0.0:8384 \
  -e TZ=Europe/Bucharest \
  -e UMASK_SET=022 \
  -p 8384:8384 \
  -p 22000:22000 \
  -p 21027:21027/udp \
  -v /mnt/Data/SyncthingTest/config:/config \
  -v /mnt/Data/SyncthingTest/data:/data1 \
  --restart unless-stopped \
  linuxserver/syncthing

Ran the script and started the container. Looking into /mnt/Data/SyncthingTest

[~]$ ls -la /mnt/Data/SyncthingTest/
total 16
drwxrwxr-x  4 pingu  pingu  4096 Jul 22 22:23 .
drwxrwxrwx 20 root   root   4096 Jul 22 22:14 ..
drwx------  4 100911 100911 4096 Jul 22 22:24 config
drwxrwxr-x  2 pingu  pingu  4096 Jul 22 22:23 data

Output of command " podman unshare ls -l ":

[pingu@pingu ~]$ podman unshare ls -l /mnt/Data/SyncthingTest/
total 8
drwx------ 4  912  912 4096 Jul 22 22:24 config
drwxrwxr-x 2 root root 4096 Jul 22 22:23 data

My take from what i see there is that "config" is used by container as it initialises and takes ownership and "data" folder is kept as root due to it not being utilised. This is cofnirmed by:

[pingu@pingu ~]$ podman unshare ls -n /mnt/Data/SyncthingTest/
total 8
drwx------ 4 912 912 4096 Jul 22 22:24 config
drwxrwxr-x 2   0   0 4096 Jul 22 22:23 data

The above test is done via rootless podman. Running as root podman as far as i understand the command "--userns=keep-id" is ignored. (http://docs.podman.io/en/latest/markdown/podman-create.1.html) I can try the test in root mode as well.
What is important to me is that the file keep an consistent GID and UID of the host user.

The environment uses more than one container on the same set of data and each having their own user and group is not ideal i think. Additionally this whole mount is shared via Samba over the network which uses GID and UID 1000.

[mheon]

I'm pretty sure that Docker does not support --userns=keep-id - so that is likely the key difference.

[gigabit942007]

I'm pretty sure that Docker does not support --userns=keep-id - so that is likely the key difference.

Corect, for Docker i would map the user ID like this:
-e PUID=1000
-e PGID=1000

However this does not work in Podman :(
I'm looking to either emulate or achieve the same in Podman.

[mheon]

Is the user running the container UID/GID 1000? The default Podman user mapping maps root in the container to the user running the container, and it sounds like that's what you're looking for?

--userns=keep-id is only required if the app in the container is run as a non-root user (things like Toolbox, which want the experience in the container to closely match the system outside the container).

[gigabit942007]

The user running the container is UID/GID 1000. I am aware of this behavior that the user running the container becomes root in the container.
Running the app inside as root would fix the issue, however is this even recomended? i would not want to run apps as root even inside containers, does not sound as a good ideea in general. I am looking for a solution where root is not the user running the app inside the container. Would there be any other solution?

[mheon]

--userns=keep-id would work as long as the app in the container runs with the UID/GID of the user running Podman.

In general, there isn't much security benefit to running as non-root in a rootless container. Root inside a rootless container isn't special - it doesn't have any privileges your user outside the container does not have. Running as a non-root user can deny access to files outside the container that are owned by your user, but then you run into situations like this - you actually want the container to be able to access files owned by your user, so we can't do that. In general, running as non-root in a rootless container is only useful if you don't need to touch files outside of the container - for something like a webapp that doesn't require anything volume-mounted in, it works perfectly fine and can improve security by a bit.

[rhatdan]

Well the security benefit, is that files in your homedir are protected from processes in the container that are not running as your UID. So running non root inside of a container is still important.

I wrote a blog on this.

https://www.redhat.com/sysadmin/rootless-podman-makes-sense

[gigabit942007]

I've read that, however it does not help me out in my case where i need to have a consistent storage area for more than one container. I think i'll just need to figure out how to properly map a fixed user group where all container users have access and just use the mask of 007 mask on all files and directories. Then even if the UID's are weird it should maybe work. I'll have to see how samba works then.
Side note aparently these docker containers don't want for some reason to start as root for some reason, can't make them do that.
Or i could just quit podman and install docker or just go CentOS with OpenShift, however that's just unknown teritory for me.

[rhatdan]

Podman running as root will work exactly the same was as Docker. The difference you are seeing is because you are running rootless podman with the user namespace. You would face the same problem is you ran Docker rootless.

Are all of the containers being run by the same user? If yes then they will all have the same UID mapped into the container and ti should not be a problem.

[mgnsk]

I've been using rootless podman with default namespace options which maps the host user to root in the container.
Haven't found a use for --userns=keep-id. The host and container UID's don't often match anyway since images can be built and run on different machines. Getting bind mounts working on a non-root podman non-root container requires an extra step on the host side.

$ podman version
Version:      2.0.2
API Version:  1
Go Version:   go1.14.4
Git Commit:   201c9505b88f451ca877d29a73ed0f1836bb96c7
Built:        Sun Jul 12 23:46:58 2020
OS/Arch:      linux/amd64

$ podman unshare cat /proc/self/uid_map
         0       1000          1
         1     100000      65536
$ podman unshare cat /proc/self/gid_map
         0       1007          1
         1     100000      65536

When the container is run as root inside the container then the bind mount works as expected. The directory does not have to exist on the image.

$ id -un
host
$ id -gn
host

$ mkdir -p ./mount
# Running as root mapped to host user.
$ podman run --rm -v "./mount:/mount" -w /mount --user root alpine touch test

$ ls -l ./mount/test
-rw-r--r-- 1 host host 0 Jul 23 13:38 ./mount/test

Now for example if the container contains a user UID 3000 then default ACL on the host side can ensure all files are kept writable by host user and the container user. The same can be done with groups.
From a quick look I couldn't find reasonable support for something like shiftfs outside of LXC.

$ mkdir -p ./mount
$ podman unshare setfacl -Rndm u:0:rwX,u:3000:rwX ./mount
$ podman unshare setfacl -Rnm u:0:rwX,u:3000:rwX ./mount

EDIT: The setfacl -n flag prevents it from setting the group execution on files (X is still necessary to enter directories).

Assuming there is an image which has a "user" with UID 3000.

$ podman run --rm -v "./mount:/mount" -w /mount -u user myalpine touch test2

$ ls -l ./mount/test2
-rw-rw-r--+ 1 102999 102999 0 Jul 23 13:39 ./mount/test2

$ getfacl ./mount/test2
# file: mount/test2
# owner: 102999
# group: 102999
user::rw-
user:host:rwx                 #effective:rw-
user:102999:rwx                 #effective:rw-
group::r-x                      #effective:r--
mask::rw-
other::r--

Non-root volumes work fine as long as the directory in the image is owned by the user. Haven't tried shared volumes.

[gigabit942007]

I've been using rootless podman with default namespace options which maps the host user to root in the container.
Haven't found a use for --userns=keep-id. The host and container UID's don't often match anyway since images can be built and run on different machines. Getting bind mounts working on a non-root podman non-root container requires an extra step on the host side.

$ podman version
Version:      2.0.2
API Version:  1
Go Version:   go1.14.4
Git Commit:   201c9505b88f451ca877d29a73ed0f1836bb96c7
Built:        Sun Jul 12 23:46:58 2020
OS/Arch:      linux/amd64

$ podman unshare cat /proc/self/uid_map
         0       1000          1
         1     100000      65536
$ podman unshare cat /proc/self/gid_map
         0       1007          1
         1     100000      65536

When the container is run as root inside the container then the bind mount works as expected. The directory does not have to exist on the image.

$ id -un
host
$ id -gn
host

$ mkdir -p ./mount
# Running as root mapped to host user.
$ podman run --rm -v "./mount:/mount" -w /mount --user root alpine touch test

$ ls -l ./mount/test
-rw-r--r-- 1 host host 0 Jul 23 13:38 ./mount/test

Now for example if the container contains a user UID 3000 then default ACL on the host side can ensure all files are kept writable by host user and the container user. The same can be done with groups.
From a quick look I couldn't find reasonable support for something like shiftfs outside of LXC.

$ mkdir -p ./mount
$ podman unshare setfacl -Rndm u:0:rwX,u:3000:rwX ./mount
$ podman unshare setfacl -Rnm u:0:rwX,u:3000:rwX ./mount

EDIT: The setfacl -n flag prevents it from setting the group execution on files (X is still necessary to enter directories).

Assuming there is an image which has a "user" with UID 3000.

$ podman run --rm -v "./mount:/mount" -w /mount -u user myalpine touch test2

$ ls -l ./mount/test2
-rw-rw-r--+ 1 102999 102999 0 Jul 23 13:39 ./mount/test2

$ getfacl ./mount/test2
# file: mount/test2
# owner: 102999
# group: 102999
user::rw-
user:host:rwx                 #effective:rw-
user:102999:rwx                 #effective:rw-
group::r-x                      #effective:r--
mask::rw-
other::r--

Non-root volumes work fine as long as the directory in the image is owned by the user. Haven't tried shared volumes.

This works wonders! It's basically what i needed the host user to maintain access to those files, the ACLs work.

I would have liked to have a feature in podman to allow this to be done easier that it was done here and maybe give the files back to the host, maybe this could work with a mount feature inside the container where a mask is applied? :( I find this way a bit too complicated for a container to just work. I think i solved my issue. The shared access not sure if it will work, but it works as it is for this purpose, common access i would just do container->container communication and be done with it.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[mgnsk]

I found another solution using bindfs. It transparently maps the files and directories. The source directory is ./code but I'll bind it to ./mount for easy removal.
$ mkdir -p ./code ./mount
$ export CONTAINER_UID=3000
$ export CONTAINER_GID=3000
$ podman unshare bindfs --map=0/${CONTAINER_UID}:@0/@${CONTAINER_GID} ./code ./mount

The effect is that host files appear as owned by container UID inside the container and new files created by the container UID will be chowned to the host user (I'm using the default podman settings of mapping host user to container root but running as another user in the container).

I'm not sure if this is possible with the podman --uidmap flag, haven't gotten it to work yet.

edit: typo

[rhatdan]

@giuseppe PTAL

[gigabit942007]

I found another solution using bindfs. It transparently maps the files and directories. The source directory is ./code but I'll bind it to ./mount for easy removal.
$ mkdir -p ./code ./mount
$ export CONTAINER_UID=3000
$ export CONTAINER_GID=3000
$ podman unshare bindfs --map=0/${CONTAINER_UID}:@0/@${CONTAINER_GID} ./code ./mount

The effect is that host files appear as owned by container UID inside the container and new files created by the container UID will be chowned to the host user (I'm using the default podman settings of mapping host user to container root but running as another user in the container).

I'm not sure if this is possible with the podman --uidmap flag, haven't gotten it to work yet.

edit: typo

This actually sounds usefull, for my usecase the solutions don't really work well as i would expect so i'm reverting to docker as that fits my bill a lot better. Will run docker in a VM and be done with it, for my purposes having the data as such it's just not worth the time and effort. I would consider this to be a basic functionality, working with data outside of the container and podman just doesn't offer a straight option for this in shared cases.

[mgnsk]

I'm also using docker in a VM to run some legacy stuff but for dev/tinkering I prefer podman.

I looked into fuse-overlayfs and it gives the same effect without chowning the files:
podman unshare fuse-overlayfs -o uidmapping=0:3000:1,gidmapping=0:3000:1,lowerdir=code,upperdir=code,workdir=workdir mount

lowerdir and upperdir are the same since I only use it for the UID shifting. The only issue is that it leaves some junk files around (which are only visible to the host). I'm guessing this could be a little more documented and integrated.

[coreyryanhanson]

@mgnsk Thank you, this solution seems to be what I was looking for! However, I don't think I have the namespaces mapped quite right yet and wanted to make sure I was actually headed in the right direction.

If one is running a container where the default account is a non-root user, will this allow writing files without both the --privileged tag and bouncing between file ownership? Am I assuming correctly that the first uid is how it appears within the container and the second the one you want to share with your host user? Or should I factor in the longer uids that are generated that the host perceives. I was also running the subsequent call to start the image within "podman unshare" does this file uid mapping remove the need to do that?

All of the namespace caveats are new to me, and I haven't found a good beginner friendly resource for a getting a strong grasp on what is happening under the hood in respect to those. Hoping this is the way that rootless containers can just work as expected without needing to invoke privileged.

[rhatdan]

This is a lot to digest, but I believe the issue is solved. Closing, reopen if I am mistaken.

[Hi-Angel]

This bug wasn't solved or am I missing something? There's a workaround of --userns=keep-id which works for me. However, it isn't default and there's doesn't seem to be an option to make that default, or is there?

This is a deal-breaker for podman-docker packages, because when you try to replace docker with podman through means of that package, you stumble upon exactly this incompatibility which breaks everything.

[rhatdan]

I really do not know what you are expecting. Docker does not have a --userns flag at all, so not sure why this is not a replacement. Show me a command in Docker which does not work the same was as Podman.

[Hi-Angel]

Show me a command in Docker which does not work the same was as Podman.

Sure

$ docker run --rm -v ~:/mnt/ -t ubuntu:20.04 /bin/ls -ln /mnt
total 0
drwxrwxr-x 3 998 998 22 Nov  7 13:47 builds
$ podman run --rm -v ~:/mnt/ -t ubuntu:20.04 /bin/ls -ln /mnt
total 0
drwxrwxr-x 3 0 0 22 Nov  7 13:47 builds

This causes problems like Permission Denied when you're compiling inside the container as a non-root user (which is a part of CI, that makes sure that build can happen as a non-root user). That works in Docker but not in Podman.

[rhatdan]

You are comparing apples and oranges. The files are owned by the same user but Podman is in a user namespace, while Docker is running as root in the hosts user namespace.

So you are compilin as user 998 within the container, which in the case of a user namespace is not the correct user.

You can create a file like
$ cat ~/.config/containers/containers.conf
[containers]
userns = "keep-id"
$ podman run --rm -v ~:/mnt/ -t ubuntu:20.04 /bin/ls -ld /mnt
drwx--x--x. 378 dwalsh dwalsh 32768 Nov 7 12:30 /mnt

[Hi-Angel]

Oh, so there is an option to make that default, the

[containers]
userns = "keep-id"

Thanks! That works for me

[Hi-Angel]

I just found an unpleasant surprise: the --userns=keep-id doesn't just keep the IDs unchanged — instead it changes the user inside the container! To a non-existing one, actually.

You are comparing apples and oranges. The files are owned by the same user but Podman is in a user namespace, while Docker is running as root in the hosts user namespace.

@rhatdan what you mention is just an implementation detail. But from the user POV there's a difference in behavior of Podman and Docker, which actually matters, which is what this bug is about. Adding the userns = "keep-id" option globally breaks containers that doesn't even try to access mount-binds: merely a ls /root/ for a unmodified ubuntu:20.04 results in Permission denied.

[mheon]

It's an implementation detail, but it is a critical part of how rootless actually works. Without user namespaces, none of this would be possible; unfortunately, that requires understanding and working with them. There is no way to accomplish what you seem to be asking for (1:1 user and group mappings for a rootless container) without completely removing the security advantages of rootless containers.

[rhatdan]

@mheon is correct, if you want to run like Docker, then you need to run as root or run your commands through sudo.

[Hi-Angel]

I just found an unpleasant surprise: the --userns=keep-id doesn't just keep the IDs unchanged — instead it changes the user inside the container! To a non-existing one, actually.

Turns out, since 4.3.0 release there is a new functional that solves that problem: the keep-id option allows to explicitly define uid/gid mapping of bind-mounts. Example: --userns=keep-id:uid=998,gid=1001. So you just pass the uid/gid of the user that is inside container, and it just works™.