User Podman service is also available

[bertmelis]

Host: Debian 10
Cockpit-podman 28-1 (build from source)

Every time I open the Podman containers page I have the option to start the "user podman service". However, this does not seem to be persistent. Every time I navigate away and return, I have to "re-enable" the user podman service:

I have no clue what it does.

[martinpitt]

What it does is that it checks/enables this:

systemctl --user status podman.socket

Is this running for you? Is it enabled? Does it somehow re-disable itself (e.g. because of some error)?

When you press the "Start" button, does that actually succeed starting podman.socket? Can you see your user images/containers?

[bertmelis]

While the "start-button" is still present:

$ systemctl --user status podman.socket
● podman.socket - Podman API Socket
   Loaded: loaded (/usr/lib/systemd/user/podman.socket; enabled; vendor preset: enabled)
   Active: active (listening) since Sat 2021-04-10 22:01:23 CEST; 13h ago
     Docs: man:podman-system-service(1)
   Listen: /run/user/1000/podman/podman.sock (Stream)
   CGroup: /user.slice/user-1000.slice/[email protected]/podman.socket

I can see my containers although CPU and memory info is not available:

No obvious error messages in the logs. But I also don't know what to look out for.

[bertmelis]

aha, a clue here:

Request Failed(Internal Server Error): failed to obtain Container 2d47da687fea3e3b1949a8d7f692a77563f8eb21fbb2889e9de3f9764ba1e3b2 stats: unable to load cgroup at /user.slice/user-1000.slice/[email protected]/user.slice/podman-32131.scope: no support for CGroups V1 in rootless environments

[marusak]

Can you please run these two command in terminal as normal user and post output here? The first one should hang (it is listening on upcoming events. Please leave it listening for ~2minutes and then you can ctrl+C it. If it does not hang that is the issue).

curl --unix-socket /run/user/1000/podman/podman.sock http://d/v1.24/events
curl --unix-socket /run/user/1000/podman/podman.sock http://d/v1.24/libpod/info

aha, a clue here:

That is expected. That is why you have n/a in the UI. For cgroupsV1 these items are just not available.

[bertmelis]

curl --unix-socket /run/user/1000/podman/podman.sock http://d/v1.24/events

{"cause":"failed to get cursor: cannot assign requested address","message":"failed to get journal cursor: failed to get cursor: cannot assign requested address","response":500}

It doesn't hang indeed. I know it's not a cockpit-podman issue but any pointers are welcome. Feel free to close the issue here.

curl --unix-socket /run/user/1000/podman/podman.sock http://d/v1.24/libpod/info

{"host":{"arch":"amd64","buildahVersion":"1.19.4","cgroupManager":"cgroupfs","cgroupVersion":"v1","conmon":{"package":"conmon: /usr/libexec/podman/conmon","path":"/usr/libexec/podman/conmon","version":"conmon version 2.0.27, commit: "},"cpus":4,"distribution":{"distribution":"debian","version":"10"},"eventLogger":"journald","hostname":"server","idMappings":{"gidmap":[{"container_id":0,"host_id":1000,"size":1},{"container_id":1,"host_id":100000,"size":65536}],"uidmap":[{"container_id":0,"host_id":1000,"size":1},{"container_id":1,"host_id":100000,"size":65536}]},"kernel":"4.19.0-16-amd64","memFree":198455296,"memTotal":16453894144,"ociRuntime":{"name":"crun","package":"crun: /usr/bin/crun","path":"/usr/bin/crun","version":"crun version 0.18.1-7931a-dirty\ncommit: 7931a1eab0590eff4041c1f74e2844b297c31cea\nspec: 1.0.0\n+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL"},"os":"linux","security":{"apparmorEnabled":false,"capabilities":"CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT","rootless":true,"seccompEnabled":true,"selinuxEnabled":false},"slirp4netns":{"executable":"/usr/bin/slirp4netns","package":"slirp4netns: /usr/bin/slirp4netns","version":"slirp4netns version 1.1.8\ncommit: unknown\nlibslirp: 4.3.1-git\nSLIRP_CONFIG_VERSION_MAX: 3\nlibseccomp: 2.4.4"},"swapFree":1001648128,"swapTotal":1024454656,"uptime":"21h 40m 50.91s (Approximately 0.88 days)","linkmode":"dynamic"},"store":{"configFile":"/home/bert/.config/containers/storage.conf","containerStore":{"number":4,"paused":0,"running":4,"stopped":0},"graphDriverName":"overlay","graphOptions":{"overlay.mount_program":{"Executable":"/usr/bin/fuse-overlayfs","Package":"fuse-overlayfs: /usr/bin/fuse-overlayfs","Version":"fusermount3 version: 3.4.1\nfuse-overlayfs: version 1.4\nFUSE library version 3.4.1\nusing FUSE kernel interface version 7.27"}},"graphRoot":"/home/bert/.local/share/containers/storage","graphStatus":{"Backing Filesystem":"extfs","Native Overlay Diff":"false","Supports d_type":"true","Using metacopy":"false"},"imageStore":{"number":34},"runRoot":"/run/user/1000/containers","volumePath":"/home/bert/.local/share/containers/storage/volumes"},"registries":{"search":["docker.io","quay.io"]},"version":{"APIVersion":"3.0.0","Version":"3.0.1","GoVersion":"go1.14","GitCommit":"","BuiltTime":"Thu Jan  1 01:00:00 1970","Built":0,"OsArch":"linux/amd64"}}

[martinpitt]

BTW, Debian 10 is not officially supported by cockpit-podman, as podman itself is not in Debian 10. I suppose you got that from some third-party repo, and that may not package the user service correctly?

[bertmelis]

That's true, I built it from source. I build cockpit-podman that is. Podman itself was installed using their guide: https://podman.io/getting-started/installation

I've enabled cgroups v2 now. Now curl --unix-socket /run/user/1000/podman/podman.sock http://d/v1.24/libpod/info hangs. But the cpu and memory are shown.

EDIT: now the command returns the info again. the only thing I remember doing meanwhile is to push the "start" buntton again.

[bertmelis]

containers/podman#8125

I'll check this tomorrow.

[marusak]

Now curl --unix-socket /run/user/1000/podman/podman.sock http://d/v1.24/libpod/info hangs. But the cpu and memory are shown.

So it hangs? Is the alert still being shown?

[bertmelis]

Nono, I'm still at square one. Only difference is that I'm using cgroups v2 now.

[marusak]

Right. I am not able to help you much more then that. If your podman is not working, you might ask in https://github.com/containers/podman . But there might be a reason why it is not in debian 10.
Also we don't test this version so it may happen that c-podman won't fully work anyway. We test on debian 11 so that should be working.

[bertmelis]

I'll check over there. Thank you for the help.
Debian Bullseye is on it's way so I might not spend too much time on this as my full rootless containers are working.

for reference: containers/podman#9993

[linickx]

Hello @marusak

I also see this, I'm on RHEL...

$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.3 (Ootpa)
$
$ rpm -qa | grep podman
podman-catatonit-2.2.1-7.module+el8.3.1+9857+68fb1526.x86_64
cockpit-podman-27.1-3.module+el8.3.1+9857+68fb1526.noarch
podman-2.2.1-7.module+el8.3.1+9857+68fb1526.x86_64
$ 

After clicking start.... I can see the service starts...

$ systemctl --user status podman.socket
● podman.socket - Podman API Socket
   Loaded: loaded (/usr/lib/systemd/user/podman.socket; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2021-05-08 16:03:47 BST; 1 day 23h ago
     Docs: man:podman-system-service(1)
   Listen: /run/user/1000/podman/podman.sock (Stream)
   CGroup: /user.slice/user-1000.slice/[email protected]/podman.socket

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
$ 

But the cockpit WebUI still "complains" all the time.

Same symptoms as above, the user containers do show up, but without CPU/Memory information.

I ran your command...

$ curl --unix-socket /run/user/1000/podman/podman.sock http://d/v1.24/events
curl: (18) transfer closed with outstanding read data remaining
$ 

and

$ curl --unix-socket /run/user/1000/podman/podman.sock http://d/v1.24/libpod/info
{"host":{"arch":"amd64","buildahVersion":"1.18.0","cgroupManager":"cgroupfs","cgroupVersion":"v1","conmon":{"package":"conmon-2.0.22-3.module+el8.3.1+9857+68fb1526.x86_64","path":"/usr/bin/conmon","version":"conmon version 2.0.22, commit: a40e3092dbe499ea1d85ab339caea023b74829b9"},"cpus":4,"distribution":{"distribution":"\"rhel\"","version":"8.3"},"eventLogger":"file","hostname":"svr.linickx.com","idMappings":{"gidmap":[{"container_id":0,"host_id":1000,"size":1},{"container_id":1,"host_id":100000,"size":65536}],"uidmap":[{"container_id":0,"host_id":1000,"size":1},{"container_id":1,"host_id":100000,"size":65536}]},"kernel":"4.18.0-240.22.1.el8_3.x86_64","memFree":460132352,"memTotal":7975350272,"ociRuntime":{"name":"runc","package":"runc-1.0.0-70.rc92.module+el8.3.1+9857+68fb1526.x86_64","path":"/usr/bin/runc","version":"runc version spec: 1.0.2-dev"},"os":"linux","rootless":true,"slirp4netns":{"executable":"/usr/bin/slirp4netns","package":"slirp4netns-1.1.8-1.module+el8.3.1+9857+68fb1526.x86_64","version":"slirp4netns version 1.1.8\ncommit: d361001f495417b880f20329121e3aa431a8f90f\nlibslirp: 4.3.1\nSLIRP_CONFIG_VERSION_MAX: 3\nlibseccomp: 2.4.3"},"swapFree":8290299904,"swapTotal":8313106432,"uptime":"189h 4m 10.55s (Approximately 7.88 days)","linkmode":"dynamic"},"store":{"configFile":"/home/linickx/.config/containers/storage.conf","containerStore":{"number":1,"paused":0,"running":1,"stopped":0},"graphDriverName":"overlay","graphOptions":{"overlay.mount_program":{"Executable":"/usr/bin/fuse-overlayfs","Package":"fuse-overlayfs-1.3.0-2.module+el8.3.1+9857+68fb1526.x86_64","Version":"fusermount3 version: 3.2.1\nfuse-overlayfs: version 1.3\nFUSE library version 3.2.1\nusing FUSE kernel interface version 7.26"}},"graphRoot":"/home/linickx/.local/share/containers/storage","graphStatus":{"Backing Filesystem":"xfs","Native Overlay Diff":"false","Supports d_type":"true","Using metacopy":"false"},"imageStore":{"number":1},"runRoot":"/run/user/1000/containers","volumePath":"/home/linickx/.local/share/containers/storage/volumes"},"registries":{"search":["registry.access.redhat.com","registry.redhat.io","docker.io"]},"version":{"APIVersion":"2","Version":"2.2.1","GoVersion":"go1.14.7","GitCommit":"","BuiltTime":"Mon Feb  8 21:19:06 2021","Built":1612819146,"OsArch":"linux/amd64"}}

Is there a way to make the "start" permanent so clicking it all the time is not needed?

[marusak]

Is there a way to make the "start" permanent so clicking it all the time is not needed?

$ curl --unix-socket /run/user/1000/podman/podman.sock http://d/v1.24/events
curl: (18) transfer closed with outstanding read data remaining
$ 

^ This is problem with podman and unfortunately there is no workaround that can be done on our side to work around it.
I think that the only "solution" would be to update to newer podman (which may unfortunately not be available for 8.3)

but without CPU/Memory information.

That is unrelated. That is due to using cgroupsV1.

[linickx]

Thanks very much 👍

So both issues are the curse of enterprise linux.

To save someone in future a google, these 2x links seem to explain why cgroupsV1 is the default on RHEL8.

• https://www.redhat.com/en/blog/world-domination-cgroups-rhel-8-welcome-cgroups-v2
• https://access.redhat.com/articles/3735611