Cannot configure rootless fuse-overlayfs globally

[iskunk]

Issue Description

I am unable to configure Podman to use fuse-overlayfs, instead of the native overlay driver, via the system-wide configuration in /etc/containers/storage.conf.

This issue occurs identically in Podman versions 3.4.4 (as shipped in Ubuntu 22.04) as well as 4.4.3 (using the Kubic packages).

I am using a rootless-only configuration. Even though my system can run the native overlay storage driver, I want Podman to use fuse-overlayfs instead. The FUSE option may normally be slower, but the current lack of idmap support for rootless native overlays makes the native driver significantly slower when using --userns=keep-id. The following references explain the situation:

• --userns=keep-id storage-chown-by-maps kills machine with large images #16541 (comment)

• --userns=keep-id storage-chown-by-maps kills machine with large images #16541 (comment)

• --userns=keep-id storage-chown-by-maps kills machine with large images #16541 (comment)

• --userns=keep-id storage-chown-by-maps kills machine on non-btrfs with large images #16830

Steps to reproduce the issue

I set the following configuration in /etc/containers/storage.conf:

[storage]
driver = "overlay"
rootless_storage_path = "/local/$USER/containers/storage"

[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"

The last line is, as I understand, the correct way of forcing the use of fuse-overlayfs despite the native driver being available.

Describe the results you received

podman info, run as a user, shows this:

  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /local/iskunk/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"

Native Overlay Diff = "true" directly contradicts the aforementioned mount_program = "/usr/bin/fuse-overlayfs" setting, even though the rootless_storage_path setting was respected. If I start a container with podman container run ... --userns=keep-id in this state, it takes an unreasonably long time to initialize due to storage-chown-by-maps.

Describe the results you expected

    Native Overlay Diff: "false"

and thus no storage-chown-by-maps slowdown at container start.

podman info output

host:
  arch: amd64
  buildahVersion: 1.29.0
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2:2.1.7-0ubuntu22.04+obs15.10_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 89.88
    systemPercent: 6.64
    userPercent: 3.48
  cpus: 1
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: file
  hostname: image-ubuntu64
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.0-67-generic
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 1509597184
  memTotal: 2072027136
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun_101:1.8.2-0ubuntu22.04+obs53.1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.2
      commit: bf70c97995109417ac0c5ed0a758d82662456724
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.0.1-2_amd64
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.6.1
  swapFree: 541061120
  swapTotal: 541061120
  uptime: 0h 2m 47.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  registry.example.com:
    Blocked: false
    Insecure: false
    Location: registry.example.com
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: registry.example.com
    PullFromMirror: ""
  search:
  - registry.example.com
store:
  configFile: /home/iskunk/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /local/iskunk/containers/storage
  graphRootAllocated: 7677796352
  graphRootUsed: 3210596352
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /local/iskunk/containers/storage/volumes
version:
  APIVersion: 4.4.3
  Built: 0
  BuiltTime: Wed Dec 31 19:00:00 1969
  GitCommit: ""
  GoVersion: go1.18.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.4.3

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Running on Ubuntu 22.04 / amd64.

Additional information

I am able to configure the use of fuse-overlayfs via ~/.config/containers/storage.conf, but cannot rely on per-user configurations at my site.

[Luap99]

@giuseppe PTAL
Looks like all options except ignore_chown_errors are ignored here: https://github.com/containers/storage/blob/b1216421c44b5383756ea3b29d6693c41f05d812/types/options.go#L298-L305
I have no idea why?

[giuseppe]

we cherry-pick some settings from the global file under /etc/containers, because that file is really meant for root not as a base for rootless users.

I don't remember why we did that for ignore_chown_errors, maybe because it was not possible to hit this issue with root anyway?

If you set the storage driver there, I expect root containers to be affected, not rootless. You need to copy the configuration for each rootless user

[iskunk]

If you set the storage driver there, I expect root containers to be affected, not rootless. You need to copy the configuration for each rootless user

The user configuration has the disadvantage that it is the same regardless of what host the user is running Podman on. The rootless_storage_path setting can naturally vary by host (by virtue of varying disk configurations), and in principle, the same could be true of mount_program.

This is on top of the administrative overhead of requiring a large number of users to set up a config file in their home directories, especially when the settings are admin-provided (i.e. there is no point to the user setting their own values), and Docker proper imposes no such burden.

If /etc/containers/storage.conf is meant for rootful Podman, then we need some other place to put system-scoped rootless settings. The system-related dependencies involved in running even rootless containers are too great to be relegated to per-user configurations alone.

[kpansky]

Unless I'm mistaken, I think you can effectively have rootless podman make use of all the settings in /etc/containers/storage.conf by using the CONTAINERS_STORAGE_CONF environment variable. For example, adding an entry into /etc/profile.d with export CONTAINERS_STORAGE_CONF=/etc/containers/storage.conf should by default have rootless users making use of all the settings in that config.

[iskunk]

Good catch, @kpansky. That appears to do the trick with 4.4.x, but is ineffectual with Ubuntu 22.04's 3.4.x. (I still see Native Overlay Diff: "true" there, and this remains the case even if I use a different filename.)

Of course, this prevents the user config file from being referenced at all, so it's not a general solution. But it does provide a less-invasive workaround for those using newer versions of Podman.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Since you have a solution, I am going to close this issue.

[iskunk]

@rhatdan, where is the root-cause bug of the mount_program keyword scope not matching rootless_storage_path being tracked?

[rhatdan]

No idea, @giuseppe do you know?

[giuseppe]

The mount program specified in the global configuration file applies to the root configuration, not rootless

[iskunk]

@giuseppe, as far as I know there is currently no way to specify a default mount program for rootless in the global config. Making the global mount_program directive cover rootless in addition to rootful would be the simplest solution, but if it really should remain rootful-only, then perhaps e.g. a new rootless_mount_program parameter would be the way to go.

In general, it should be possible for the global config to specify the default for any user-level rootless parameter, which the user config can then override---i.e. standard administrative hierarchy. When a parameter only works in the user config, it makes things difficult for the sysadmin, who now has to reach into the user configs in some way to set that parameter (if it's one that is needed to get the installation working). The above issue is a specific instance of this more general problem of rootless config scope.

Is there an existing issue tracking the general problem? I can create one if needed.