Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dbus-launch and conmon/pids.max problem #13382

Closed
xfoobar opened this issue Mar 1, 2022 · 7 comments · Fixed by #13398
Closed

dbus-launch and conmon/pids.max problem #13382

xfoobar opened this issue Mar 1, 2022 · 7 comments · Fixed by #13398
Assignees
@giuseppe
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@xfoobar
Copy link

xfoobar commented Mar 1, 2022 •
edited
Loading

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Error warning when testing pod, dbus-launch and conmon/pids.max

Steps to reproduce the issue:

podman pod create --name=foo && podman pod rm foo

Describe the results you received:

1663c402dc20339d604a65b2e282bbdf64e251682f4347515ea6c8c56ca7f7eb
WARN[0000] Failed to add pause process to systemd sandbox cgroup: exec: "dbus-launch": executable file not found in $PATH
WARN[0000] Error updating pod 1663c402dc20339d604a65b2e282bbdf64e251682f4347515ea6c8c56ca7f7eb conmon cgroup PID limit: open /sys/fs/cgroup/libpod_parent/1663c402dc20339d604a65b2e282bbdf64e251682f4347515ea6c8c56ca7f7eb/conmon/pids.max: no such file or directory
1663c402dc20339d604a65b2e282bbdf64e251682f4347515ea6c8c56ca7f7eb

Describe the results you expected:

1663c402dc20339d604a65b2e282bbdf64e251682f4347515ea6c8c56ca7f7eb
1663c402dc20339d604a65b2e282bbdf64e251682f4347515ea6c8c56ca7f7eb

Additional information you deem important (e.g. issue happens only occasionally):
Build from source on Debian 11 without GUI, go1.17.7 linux/amd64.

sudo apt install conmon crun rootlesskit  pkg-config libseccomp-dev  build-essential  libbtrfs-dev libgpgme-dev libdevmapper-dev git -y

cd podman-4.0.1
make
sudo make install
sudo make install.completions

create file /etc/containers/policy.json

{
    "default": [
        {
            "type": "insecureAcceptAnything"
        }
    ],
    "transports":
        {
            "docker-daemon":
                {
                    "": [{"type":"insecureAcceptAnything"}]
                }
        }
}

create file ~/.config/containers/containers.conf

[containers]

[engine]
infra_image = "kubernetes/pause:latest"

[machine]

[network]

[secrets]

Output of podman version:

Client:       Podman Engine
Version:      4.0.1
API Version:  4.0.1
Go Version:   go1.17.7

Built:      Mon Feb 28 22:18:18 2022
OS/Arch:    linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 8
  distribution:
    codename: bullseye
    distribution: debian
    version: "11"
  eventLogger: file
  hostname: service
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.10.0-11-amd64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 3748442112
  memTotal: 4120764416
  networkBackend: cni
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.4.0
  swapFree: 0
  swapTotal: 0
  uptime: 12m 24.5s
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  docker.io:
    Blocked: false
    Insecure: false
    Location: xxxxxx.mirror.aliyuncs.com
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: docker.io
  search:
  - docker.io
store:
  configFile: /home/foobar/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/foobar/.local/share/containers/storage
  graphStatus: {}
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/foobar/.local/share/containers/storage/volumes
version:
  APIVersion: 4.0.1
  Built: 1646104698
  BuiltTime: Mon Feb 28 22:18:18 2022
  GitCommit: ""
  GoVersion: go1.17.7
  OsArch: linux/amd64
  Version: 4.0.1

Package info (e.g. output of rpm -q podman or apt list podman):

None, Build from source.

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.): physical
@openshift-ci openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Mar 1, 2022
@mheon
Copy link
Member

mheon commented Mar 1, 2022

WARN[0000] Failed to add pause process to systemd sandbox cgroup: exec: "dbus-launch": executable file not found in $PATH

This seems a bit concerning, as podman info says this is a cgroupfs system - we shouldn't be using systemd cgroups in that case. @giuseppe PTAL

The other error indicates we're unable to set pids.max on the pod cgroup, which we do in order to prevent cleanup processes from spawning and complicating teardown of the pod. That's a little concerning, but probably an environment issue?

@giuseppe
Copy link
Member

giuseppe commented Mar 1, 2022

that fails because rootless cannot use cgroupfs on cgroupv2, unless it is configured manually.

IMO, the best for both warnings is to downgrade them to debug when running rootless.

@xfoobar
Copy link
Author

xfoobar commented Mar 2, 2022 •
edited
Loading

@giuseppe
I reinstalled Debian 11 , there are some changes in the output.

podman pod create --name=foo && podman pod rm foo

9c3b72e4a8d1f4d148ba3f5e80699594b8cb2fe387f2cb8dd4d19ab3927ceec5
WARN[0000] Error updating pod 9c3b72e4a8d1f4d148ba3f5e80699594b8cb2fe387f2cb8dd4d19ab3927ceec5 conmon cgroup PID limit: open /sys/fs/cgroup/libpod_parent/9c3b72e4a8d1f4d148ba3f5e80699594b8cb2fe387f2cb8dd4d19ab3927ceec5/conmon/pids.max: no such file or directory
9c3b72e4a8d1f4d148ba3f5e80699594b8cb2fe387f2cb8dd4d19ab3927ceec5

podman info --debug

host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 8
  distribution:
    codename: bullseye
    distribution: debian
    version: "11"
  eventLogger: file
  hostname: debian
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.10.0-11-amd64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 649084928
  memTotal: 2077491200
  networkBackend: cni
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.4.0
  swapFree: 1021571072
  swapTotal: 1022357504
  uptime: 26m 26.93s
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  docker.io:
    Blocked: false
    Insecure: false
    Location: xxxxxx.mirror.aliyuncs.com
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: docker.io
  search:
  - docker.io
store:
  configFile: /home/foobar/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/foobar/.local/share/containers/storage
  graphStatus: {}
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/foobar/.local/share/containers/storage/volumes
version:
  APIVersion: 4.0.1
  Built: 1646189740
  BuiltTime: Tue Mar  1 21:55:40 2022
  GitCommit: ""
  GoVersion: go1.17.7
  OsArch: linux/amd64
  Version: 4.0.1

giuseppe added a commit to giuseppe/libpod that referenced this issue Mar 2, 2022
@giuseppe
libpod: drop warning if cgroup doesn't exist
e71d497 
do not print a warning on cgroup removal if it doesn't exist.

Closes: containers#13382

[NO NEW TESTS NEEDED]

Signed-off-by: Giuseppe Scrivano <[email protected]>
@giuseppe giuseppe mentioned this issue Mar 2, 2022
Merged
@giuseppe
Copy link
Member

giuseppe commented Mar 2, 2022

opened a PR: #13398

giuseppe added a commit to giuseppe/libpod that referenced this issue Mar 2, 2022
@giuseppe
libpod: drop hack to set conmon cgroup pids.max=1
c4472e1 
avoid forcing the pids.max = 1 limit to avoid cleanup processes, which
is racy since the cleanup processes could be triggered by the
container exiting; and it doesn't work with rootless when it cannot
use cgroups, i.e. cgroupfs and cgroup v1).

Closes: containers#13382

[NO NEW TESTS NEEDED] it doesn't add any new functionality

Signed-off-by: Giuseppe Scrivano <[email protected]>
@giuseppe giuseppe mentioned this issue Mar 2, 2022
Closed
@giuseppe
Copy link
Member

giuseppe commented Mar 2, 2022

alternative fix: #13403

@giuseppe
Copy link
Member

@mheon are you fine with #13398 or #13403?

@mheon
Copy link
Member

mheon commented Mar 22, 2022

Preference would be for #13398 but if folks prefer 13403 I will not object too strongly

mheon pushed a commit to mheon/libpod that referenced this issue Mar 30, 2022
@giuseppe @mheon
libpod: drop warning if cgroup doesn't exist
4dbb2bf 
do not print a warning on cgroup removal if it doesn't exist.

Closes: containers#13382

[NO NEW TESTS NEEDED]

Signed-off-by: Giuseppe Scrivano <[email protected]>
@giuseppe giuseppe mentioned this issue Apr 29, 2022
Merged
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 20, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 20, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@giuseppe giuseppe

Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Milestone
No milestone
3 participants
@giuseppe @mheon @xfoobar