v4.1.0 User volumes are not mounted when default volumes are configured in /etc/container/containers.conf

[r10r]

/kind bug

Description

In podman v4.1.0 user volumes (podman run -v) or mounts (podman run --mount) are not added to the OCI spec and not mounted when default volumes are configured in /etc/containers/containers.conf e.g

volumes = ["/etc/ssl/certs:/etc/ssl/certs:ro"]

In podman v4.0.3 user mounts and default volumes added to the OCI spec and mounted.

Steps to reproduce the issue:

1. Set volumes = ["/etc/ssl/certs:/etc/ssl/certs:ro"] in /etc/containers/containers.conf

2. Create a pod with a container that has additional user mounts

I used the following script for testing.

#!/bin/sh -eu

pod=mount-test

cleanup() {
    rmdir $tmproot
    # delay cleanup else `pod rm` might fail
    sleep 3
    $PODMAN pod rm $pod >/dev/null
}

$PODMAN pod create --name $pod >/dev/null
tmproot=$(mktemp -d)
trap cleanup EXIT
$PODMAN run -it --pod $pod -v $tmproot:/root docker.io/library/alpine:latest mount | egrep '(/root|/etc/ssl/certs)'
#$PODMAN run -it --pod $pod --mount type=bind,src=$tmproot,target=/root docker.io/library/alpine:latest mount

Describe the results you received:

The output of the test script when using v4.1.0 is

(none):~# PODMAN=podman-4.1.0 ./test-mount.sh
rootfs on /etc/ssl/certs type rootfs (ro,size=2957428k,nr_inodes=739357,inode64)

NOTE when volumes are removed from /etc/containers/containers.conf then the output is

(none):~# PODMAN=podman-4.1.0 ./test-mount.sh
rootfs on /root type rootfs (rw,size=2957428k,nr_inodes=739357,inode64)

Describe the results you expected:

The output of the test script when using v4.0.3 is

(none):~# PODMAN=podman-4.0.3 ./test-mount.sh
rootfs on /root type rootfs (rw,size=2957428k,nr_inodes=739357,inode64)
rootfs on /etc/ssl/certs type rootfs (ro,size=2957428k,nr_inodes=739357,inode64)

Additional information you deem important (e.g. issue happens only occasionally):

We did a git bisect and identified 7a53428 as first bad commit.

I looked at the commit and it seems that the Mounts fields that contains the user mounts field gets overridden by the field

7a53428#diff-5a040c4acc8459d3378715e9af985864e25b780aa6c796170bff095b35bf1883R417

from the json unmarshaller in the Inherit function here

7a53428#diff-3ff238a49e99bd584fa865688b0fd9914acd5cdd71f46606c6750a855da6cf4fR561

I wonder if this does affect other user options too?

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

[Luap99]

Thanks for the detailed report @r10r.

@cdoern PTAL

[cdoern]

weird, I thought I fixed this exact issue a month or so back. I will look at this today.

edit: I see, I think the pod is just overriding with the volumes it set, since pod volumes were broken up until 7a53428 this came out of the works then.

[cdoern]

@r10r have you tried this on the main branch, running your exact commands in rootless podman works for me. Are you sure /etc/containers/containers.conf (rootful) is the right conf file not $HOME/.config/containers/containers.conf (rootless)?

I added the same volume config to my $HOME/.config/containers/containers.conf

[charliedoern@fedora podman]$ bin/podman --version
podman version 4.2.0-dev
[charliedoern@fedora podman]$ bin/podman run -it --pod=123 -v /tmp:/tmp/123/456 alpine
/ # mount | egrep '(/tmp/123/456|/etc/ssl/certs)'
/dev/nvme0n1p3 on /etc/ssl/certs type btrfs (ro,seclabel,relatime,compress=zstd:1,ssd,space_cache,subvolid=257,subvol=/root)
tmpfs on /tmp/123/456 type tmpfs (rw,seclabel,nosuid,nodev,size=32732968k,nr_inodes=1048576,inode64)

[r10r]

@cdoren I tried it with the main branch, but same issue. I'm running podman rootful.

Please see the debug information below. I've additionally attached the debug output of the command PODMAN=/mnt/rw/plain/dev/podman-main ./test-mount.sh > podman-run-debug.txt 2>&1

podman-run-debug.txt

Btw. I've build podman using the following Dockerfile

FROM docker.io/library/golang:1.17-buster as build
ARG DEBIAN_FRONTEND="noninteractive"
RUN apt-get update \
        && apt-get install -y seccomp libseccomp-dev \
        && apt-get clean \
        && rm -rf /var/lib/apt/lists/*

WORKDIR /workdir
ADD https://github.com/containers/podman/archive/refs/heads/main.tar.gz src.tar.gz
RUN tar --strip-components=1 -xf src.tar.gz
ARG BUILD_TAGS="osusergo,netgo,exclude_graphdriver_devicemapper,exclude_graphdriver_btrfs,containers_image_openpgp,seccomp"
# NOTE watch the project Makefile closely for changes
RUN go build -o /usr/local/bin/podman \
        -ldflags "-w -s -extldflags=-static -X github.com/containers/podman/v4/libpod/define.buildInfo=$(date +%s)" \
        -tags "${BUILD_TAGS}" \
        ./cmd/podman
RUN test -x /usr/local/bin/podman && ! ldd /usr/local/bin/podman

FROM docker.io/library/alpine:latest
COPY --from=build /usr/local/bin/podman /usr/local/bin/podman

The output of podman info

host:
  arch: amd64
  buildahVersion: 1.27.0-dev
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.30-r0
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: unknown'
  cpuUtilization:
    idlePercent: 99.08
    systemPercent: 0.22
    userPercent: 0.7
  cpus: 4
  distribution:
    distribution: alpine
    version: 3.15.2
  eventLogger: file
  hostname: (none)
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.16.0-5-amd64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 1120952320
  memTotal: 6220759040
  networkBackend: cni
  ociRuntime:
    name: runc
    package: Unknown
    path: /usr/local/bin/runc
    version: |-
      runc version 1.1.0
      commit: v1.1.0-0-g067aaf85
      spec: 1.0.2-dev
      go: go1.17.6
      libseccomp: 2.5.3
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 145h 16m 46.00s (Approximately 6.04 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 3
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 103609503744
  graphRootUsed: 57247027200
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1028
  runRoot: /var/lib/containers/run
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.2.0-dev
  Built: 1654598812
  BuiltTime: Tue Jun  7 12:46:52 2022
  GitCommit: ""
  GoVersion: go1.17.11
  Os: linux
  OsArch: linux/amd64
  Version: 4.2.0-dev