Running podman container with --privileged doesn't mount --device

[timmydo]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When running with the --privileged flag, I don't see the device inside the container. I don't see an error either.

Steps to reproduce the issue:

> podman run --rm -ti --privileged --device /dev/kvm alpine:latest ls -l /dev/kvm
ls: /dev/kvm: No such file or directory

> podman run --rm -ti  --device /dev/kvm alpine:latest ls -l /dev/kvm 
crw-rw----    1 nobody   nobody     10, 232 Dec 24 05:00 /dev/kvm

> podman run --rm -ti  -v /dev/kvm alpine:latest ls -l /dev/kvm 
total 0

> podman run --rm -ti  -v /dev/kvm:/dev/kvm alpine:latest ls -l /dev/kvm
crw-rw----    1 nobody   nobody     10, 232 Dec 24 05:00 /dev/kvm

> podman run --rm -ti --privileged -v /dev/kvm:/dev/kvm alpine:latest ls -l /dev/kvm
crw-rw----    1 nobody   nobody     10, 232 Dec 24 05:00 /dev/kvm

Describe the results you received:

/dev/kvm missing

Describe the results you expected:

/dev/kvm present or error message

Additional information you deem important (e.g. issue happens only occasionally):

happens every time.

Output of podman version:

 podman version
Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.14.15
Built:        Wed Dec 31 16:00:01 1969
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /gnu/store/99lswyq2y7knb80m9qm1kmid4s0fnxaq-conmon-2.0.30/bin/conmon
    version: 'conmon version 2.0.30, commit: unknown'
  cpus: 16
  distribution:
    distribution: guix
    version: unknown
  eventLogger: file
  hostname: t5700g
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 998
      size: 1
    - container_id: 1
      host_id: 1001000000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1001000000
      size: 1000000
  kernel: 5.15.10
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 60466466816
  memTotal: 66867933184
  ociRuntime:
    name: crun
    package: Unknown
    path: /gnu/store/laxnk1inyc7alvq0q61b4357jshfg37l-crun-1.3/bin/crun
    version: |-
      crun version UNKNOWN
      commit: 8e5757a4e68590326dafe8a8b1b4a584b10a1370
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /home/timmy/run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /gnu/store/lia59div8rynw1inqrprkj15czsqxzfv-slirp4netns-1.1.12/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.1.12
      commit: unknown
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 39h 1m 36.28s (Approximately 1.62 days)
plugins:
  log:
  - k8s-file
  - none
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/timmy/.config/containers/storage.conf
  containerStore:
    number: 10
    paused: 0
    running: 0
    stopped: 10
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/timmy/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 2
  runRoot: /home/timmy/run/containers
  volumePath: /home/timmy/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.4
  Built: 1
  BuiltTime: Wed Dec 31 16:00:01 1969
  GitCommit: ""
  GoVersion: go1.14.15
  OsArch: linux/amd64
  Version: 3.4.4

Package info (e.g. output of rpm -q podman or apt list podman):

I build this myself for guix. https://debbugs.gnu.org/cgi/bugreport.cgi?bug=52174
Currently at PATCH v3.

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

I looked but didn't see anything.

Additional environment details (AWS, VirtualBox, physical, etc.):

physical on GNU Guix distro.

[rhatdan]

I don't see this when I test, and it would be very strange since in rootless mode. we are just bind mounting /dev:/dev
when you run in --privileged mode.

$ podman -v
podman version 3.4.4
$ podman run --privileged fedora ls /dev/kvm
/dev/kvm
$  podman run --rm -ti --privileged --device /dev/kvm alpine:latest ls -l /dev/kvm
Resolved "alpine" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob 59bf1c3509f3 done  
Copying config c059bfaa84 done  
Writing manifest to image destination
Storing signatures
crw-rw-rw-    1 nobody   nobody     10, 232 Dec 23 11:19 /dev/kvm

[timmydo]

Do you have any recommendations for something I could do to root cause it? I don't have much experience with container internals so I don't know where to start.

> podman run --rm -ti --privileged --device /dev/kvm alpine:latest ls -l /dev/kvm
ls: /dev/kvm: No such file or directory

> podman run --rm -ti --privileged -v /dev/kvm:/dev/kvm alpine:latest ls -l /dev/kvm
crw-rw----    1 nobody   nobody     10, 232 Dec 24 05:00 /dev/kvm

[rhatdan]

@giuseppe Ideas?
On the host what does
ls -l /dev/kvm show?

[rhatdan]

ls /dev
versus
podman run fedora ls /dev/

[timmydo]

on the host:

> ls -l /dev/kvm 
crw-rw---- 1 root kvm 10, 232 Dec 23 21:00 /dev/kvm

> groups
users libvirt kvm netdev audio video input wheel

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[timmydo]

@rhatdan / @giuseppe ideas on how I can diagnose?

[giuseppe]

can you show me the mount table on the host (cat /proc/self/mountinfo)?

$ podman run --rm -ti --privileged alpine:latest ls -l /dev/kvm
crw-rw-rw-    1 nobody   nobody     10, 232 Jan 26 16:18 /dev/kvm

If you run just --privileged without --device, does it work?

[giuseppe]

I could reproduce locally, I'll update shortly

[giuseppe]

opened a PR here: #13031

[timmydo]

Thanks! I just tried the latest git (the fake 3.4.4.1 placeholder) and it works

 /gnu/store/3fh4q6rlmm5vadx7j6yp9vad9ddvikfg-podman-3.4.4.1/bin/podman run --rm -ti --privileged --device /dev/kvm alpine:latest ls -l /dev/kvm
crw-rw----    1 nobody   nobody     10, 232 Jan 24 02:36 /dev/kvm