podman commands leak dbus-daemon processes #9727

sdhoward · 2021-03-16T19:51:06Z

/kind bug

Description

When rootless podman commands run, they spawn a dbus-daemon process which stays running. Appears similar to #4483.

Steps to reproduce the issue:

set settings in containers.conf

[engine]
runtime = "/usr/bin/runc"
cgroup_manager = "cgroupfs"

run podman ps
run pgrep -a dbus-daemon to see the new process

$ pgrep -a dbus-daemon
1283 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only

$ podman ps
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES

$ pgrep -a dbus-daemon
1283 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
1053394 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session

Describe the results you received/expected:

I expect the process to exit as the container is cleaned up.

Output of podman version:

$ podman version
Version:      3.0.1
API Version:  3.0.0
Go Version:   go1.15.10
Git Commit:   c640670e85c4aaaff92741691d6a854a90229d8d
Built:        Tue Mar 16 11:24:42 2021
OS/Arch:      linux/ppc64le

Output of podman info --debug:

host:
  arch: ppc64le
  buildahVersion: 1.19.4
  cgroupManager: cgroupfs
  cgroupVersion: v2   
  conmon:
    package: app-emulation/conmon-2.0.22
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.22, commit: 9c34a8663b85e479e0c083801e89a2b2835228ed'
  cpus: 32
  distribution:
    distribution: gentoo
    version: unknown  
  eventLogger: journald
  hostname: gcom
  idMappings:
    gidmap:
    - container_id: 0 
      host_id: 1000   
      size: 1
    - container_id: 1 
      host_id: 100000 
      size: 65536
    uidmap:
    - container_id: 0 
      host_id: 1000   
      size: 1
    - container_id: 1 
      host_id: 100000 
      size: 65536
  kernel: 5.10.19-gentoo
  linkmode: dynamic   
  memFree: 6492258304 
  memTotal: 34034876416
  ociRuntime:
    name: crun
    package: app-emulation/crun-0.17
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: app-emulation/slirp4netns-1.1.9
    version: |-
      slirp4netns version 1.1.9
      commit: 4e37ea557562e0d7a64dc636eff156f64927335e
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.4
  swapFree: 68719411200
  swapTotal: 68719411200
  uptime: 273h 46m 3.88s (Approximately 11.38 days)
registries:
  search:
  - docker.io
store:
  configFile: /home/scott/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: sys-fs/fuse-overlayfs-1.4.0
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.4
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/scott/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 23
  runRoot: /run/user/1000/containers
  volumePath: /home/scott/.local/share/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 1615919082
  BuiltTime: Tue Mar 16 11:24:42 2021
  GitCommit: c640670e85c4aaaff92741691d6a854a90229d8d
  GoVersion: go1.15.10
  OsArch: linux/ppc64le
  Version: 3.0.1

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Gentoo ppc64le on Talos

The text was updated successfully, but these errors were encountered:

mheon · 2021-03-17T19:54:38Z

@giuseppe PTAL

giuseppe · 2021-03-18T07:55:02Z

we had the same issue in the past and it was addressed with #6569

Could you attach the log when you run with --log-level debug so we can check where the scope is created?

sdhoward · 2021-03-19T21:42:26Z

relevant output:

DEBU[0000] using runtime "/usr/bin/crun"                
INFO[0000] Setting parallel job count to 97             
DEBU[0000] Failed to add podman to systemd sandbox cgroup: dbus: authentication failed

any suggestion for this? processes should not continue running after podman exits, regardless, would you agree?

github-actions · 2021-04-19T00:07:07Z

A friendly reminder that this issue had no activity for 30 days.

sdhoward · 2021-06-18T00:26:10Z

this issue has no activity because I got no response, not because it's irrelevant...

mheon · 2021-06-18T15:00:35Z

Can you provide the full logs? I'm not sure if that's actually related - the auth failure is interesting, but that is not the only place we use dbus.

sdhoward · 2021-07-07T00:38:28Z

with runc:

INFO[0000] podman filtering at log level debug
DEBU[0000] Called ps.PersistentPreRunE(podman ps --log-level=debug)
DEBU[0000] overlay storage already configured with a mount-program
DEBU[0000] Merged system config "/home/user/.config/containers/containers.conf"
DEBU[0000] overlay storage already configured with a mount-program
DEBU[0000] Using conmon: "/usr/libexec/podman/conmon"
DEBU[0000] Initializing boltdb state at /home/user/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/user/.local/share/containers/storage
DEBU[0000] Using run root /run/user/1000/containers
DEBU[0000] Using static dir /home/user/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp
DEBU[0000] Using volume path /home/user/.local/share/containers/storage/volumes
DEBU[0000] overlay storage already configured with a mount-program
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false
DEBU[0000] Initializing event backend journald
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/runc"
DEBU[0000] Default CNI network name podman is unchangeable
INFO[0000] Setting parallel job count to 97
DEBU[0000] Failed to add podman to systemd sandbox cgroup: dbus: authentication failed
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES
DEBU[0000] Called ps.PersistentPostRunE(podman ps --log-level=debug)

with crun:

INFO[0000] podman filtering at log level debug
DEBU[0000] Called ps.PersistentPreRunE(podman ps --log-level=debug)
DEBU[0000] overlay storage already configured with a mount-program
DEBU[0000] Merged system config "/home/user/.config/containers/containers.conf"
DEBU[0000] overlay storage already configured with a mount-program
DEBU[0000] Using conmon: "/usr/libexec/podman/conmon"
DEBU[0000] Initializing boltdb state at /home/user/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/user/.local/share/containers/storage
DEBU[0000] Using run root /run/user/1000/containers
DEBU[0000] Using static dir /home/user/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp
DEBU[0000] Using volume path /home/user/.local/share/containers/storage/volumes
DEBU[0000] overlay storage already configured with a mount-program
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false
DEBU[0000] Initializing event backend journald
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
DEBU[0000] Default CNI network name podman is unchangeable
INFO[0000] Setting parallel job count to 97
DEBU[0000] Failed to add podman to systemd sandbox cgroup: dbus: authentication failed
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES
DEBU[0000] Called ps.PersistentPostRunE(podman ps --log-level=debug)

rhatdan · 2021-07-08T11:04:46Z

@giuseppe WDYT?

github-actions · 2021-08-08T00:03:40Z

A friendly reminder that this issue had no activity for 30 days.

sdhoward · 2021-08-10T23:10:11Z

just to fix the label: this issue is persistent and not stale

sdhoward · 2021-08-23T19:37:59Z

still having this issue in podman 3.2.1

giuseppe · 2021-08-24T11:14:35Z

I think the leak happens in the godbus library: https://github.com/godbus/dbus/blob/master/conn_other.go#L19-L38

The solution would be to make sure DBUS_SESSION_BUS_ADDRESS is defined in your environment.

I've opened an issue with godbus: godbus/dbus#268

dolph · 2021-09-14T18:30:03Z

The solution would be to make sure DBUS_SESSION_BUS_ADDRESS is defined in your environment.

On CentOS 7, I found that DBUS_SESSION_BUS_PID also had to be defined (otherwise the dbus-daemon processes would pile up), so we used this in ~/.bashrc:

hash dbus-launch 2>/dev/null && [ -z "$DBUS_SESSION_BUS_PID" ] && export $(dbus-launch)

github-actions · 2021-10-15T00:03:58Z

A friendly reminder that this issue had no activity for 30 days.

rhatdan · 2021-10-15T14:12:59Z

Since this is not a Podman bug, closing.

do not start up a dbus daemon if it is not already running. [NO NEW TESTS NEEDED] the fix is in a dependency. Closes: containers#9727 Signed-off-by: Giuseppe Scrivano <[email protected]>

giuseppe · 2021-10-26T19:08:52Z

fixed with #12107

do not start up a dbus daemon if it is not already running. [NO NEW TESTS NEEDED] the fix is in a dependency. Closes: containers#9727 Signed-off-by: Giuseppe Scrivano <[email protected]>

openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Mar 16, 2021

github-actions bot added the stale-issue label Apr 19, 2021

github-actions bot removed the stale-issue label Jun 30, 2021

github-actions bot added the stale-issue label Aug 8, 2021

rhatdan removed the stale-issue label Aug 9, 2021

rhatdan assigned giuseppe Aug 9, 2021

giuseppe mentioned this issue Aug 24, 2021

dbus-launch leaks a process godbus/dbus#268

Closed

github-actions bot added the stale-issue label Oct 15, 2021

rhatdan closed this as completed Oct 15, 2021

giuseppe reopened this Oct 26, 2021

giuseppe mentioned this issue Oct 26, 2021

cgroups: use SessionBusPrivateNoAutoStartup #12107

Merged

openshift-merge-robot closed this as completed in #12107 Oct 27, 2021

larrycai mentioned this issue Sep 6, 2022

[Fedora 33, rootless] podman pod create fails #8776

Closed

chrishecker mentioned this issue Oct 28, 2022

launching aardvark-dns with systemd user on centos7 fails due to session dbus perms? containers/netavark#473

Closed

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 21, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

podman commands leak dbus-daemon processes #9727

podman commands leak dbus-daemon processes #9727

sdhoward commented Mar 16, 2021

mheon commented Mar 17, 2021

giuseppe commented Mar 18, 2021

sdhoward commented Mar 19, 2021

github-actions bot commented Apr 19, 2021

sdhoward commented Jun 18, 2021

mheon commented Jun 18, 2021

sdhoward commented Jul 7, 2021

rhatdan commented Jul 8, 2021

github-actions bot commented Aug 8, 2021

sdhoward commented Aug 10, 2021

sdhoward commented Aug 23, 2021

giuseppe commented Aug 24, 2021

dolph commented Sep 14, 2021

github-actions bot commented Oct 15, 2021

rhatdan commented Oct 15, 2021

giuseppe commented Oct 26, 2021

podman commands leak dbus-daemon processes #9727

podman commands leak dbus-daemon processes #9727

Comments

sdhoward commented Mar 16, 2021

mheon commented Mar 17, 2021

giuseppe commented Mar 18, 2021

sdhoward commented Mar 19, 2021

github-actions bot commented Apr 19, 2021

sdhoward commented Jun 18, 2021

mheon commented Jun 18, 2021

sdhoward commented Jul 7, 2021

rhatdan commented Jul 8, 2021

github-actions bot commented Aug 8, 2021

sdhoward commented Aug 10, 2021

sdhoward commented Aug 23, 2021

giuseppe commented Aug 24, 2021

dolph commented Sep 14, 2021

github-actions bot commented Oct 15, 2021

rhatdan commented Oct 15, 2021

giuseppe commented Oct 26, 2021