Error: could not get runtime: please update to v2.0.1 or later: outdated conmon version (CentOS 8)

[svenvogel]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

1. podman-1.6.4 -> 94b7842
2. conmon 2.0.3 -> eb5fa88c26fde5ce1e3f8a1d2a8a9498b2d7dbe6

/kind feature

Description
i build podman on centos 8. it works but after install of the RPM and run an container with podman i get the following error. i used 2.0.3 of conmon.

Error: could not get runtime: please update to v2.0.1 or later: outdated conmon version

there are any hint or anybody can help whats wrong? maybe it has something to do with #4465 but i dont know...

Package info (e.g. output of rpm -q podman or apt list podman):
podman-1.6.4-1575906879.git94b78425.el8.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):
virtual Machine on KVM

[mheon]

Where is Conmon installed to? It is possible you have another Conmon installed that is being used instead of the one you've built - check to see if there are any other Conmon binaries installed at the paths listed in conmon_path in libpod.conf.

[mheon]

In particular, I would check /usr/libexec/podman/conmon - I believe we used to install a bundled Conmon there, while new Conmon installs likely choose /usr/bin

[haircommander]

you can also find the conmon that podman is choosing with podman info or running with --log-level debug

[mheon]

Part of me thinks that podman info will also explode...

We may want to put the path we're using in the error message, to make it clearer what's going on

[haircommander]

when searching for a good candidate, we actually go through every path configured and choose only the conmons that are of the right version. this message happens after each spot is searched. @svenvogel, where is the conmon 2.0.3 installed?

[svenvogel]

@haircommander thanks for the answer.

podman info gives me the same
Error: could not get runtime: please update to v2.0.1 or later: outdated conmon version

with debug
DEBU[0000] Reading configuration file "/usr/share/containers/libpod.conf" DEBU[0000] Merged system config "/usr/share/containers/libpod.conf": &{{false false false false false true} 0 { [] [] []} docker:// runc map[runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc] crun:[/usr/bin/crun /usr/local/bin/crun]] [crun runc] [crun] [] [/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] systemd /var/run/libpod -1 false /etc/cni/net.d/ [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] podman [] k8s.gcr.io/pause:3.1 /pause false false 2048 shm false} WARN[0000] Conmon at /usr/libexec/podman/conmon invalid: outdated conmon version ERRO[0000] could not get runtime: please update to v2.0.1 or later: outdated conmon version

what do you mean where are the conmon installed? i installed the podman rpm and thats it...

[svenvogel]

@haircommander @mheon

its true a old version. :(

/usr/libexec/podman/conmon --version conmon version 1.0.0-rc2 commit: f126ccedd1a9b2bc5af7cd656cc70fd1d28392f9-dirty

[haircommander]

@lsm5 do you maintain the CentOS 8 package?

[lsm5]

@lsm5 do you maintain the CentOS 8 package?

this is basically a rebuild of the rhel8 package. @jnovy ^

[svenvogel]

@lsm5 @haircommander i tried libpod master and now it works.

/usr/libexec/podman/conmon --version conmon version 2.0.3 commit: 932fe3550df5b7fd77c4288d3ebc384212bcce1d-dirty

podman --version podman version 1.6.4-dev

podman info --log-level debug DEBU[0000] Reading configuration file "/usr/share/containers/libpod.conf" DEBU[0000] Merged system config "/usr/share/containers/libpod.conf": &{{false false false false false true} 0 { [] [] []} docker:// runc map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata-runtime:[/usr/bin/kata-runtime] kata-qemu:[/usr/bin/kata-qemu] kata-fc:[/usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] [crun runc] [crun] [] [/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] systemd /var/run/libpod -1 false /etc/cni/net.d/ [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] podman [] k8s.gcr.io/pause:3.1 /pause false false 2048 shm false} DEBU[0000] Using conmon: "/usr/libexec/podman/conmon" DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver overlay DEBU[0000] Using graph root /var/lib/containers/storage DEBU[0000] Using run root /var/run/containers/storage DEBU[0000] Using static dir /var/lib/containers/storage/libpod DEBU[0000] Using tmp dir /var/run/libpod DEBU[0000] Using volume path /var/lib/containers/storage/volumes DEBU[0000] Set libpod namespace to "" DEBU[0000] [graphdriver] trying provided driver "overlay" DEBU[0000] cached value indicated that overlay is supported DEBU[0000] cached value indicated that metacopy is not being used DEBU[0000] cached value indicated that native-diff is usable DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false DEBU[0000] Initializing event backend file WARN[0000] Error initializing configured OCI runtime kata-fc: no valid executable found for OCI runtime kata-fc: invalid argument DEBU[0000] using runtime "/usr/bin/runc" WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument WARN[0000] Error initializing configured OCI runtime kata-runtime: no valid executable found for OCI runtime kata-runtime: invalid argument WARN[0000] Error initializing configured OCI runtime kata-qemu: no valid executable found for OCI runtime kata-qemu: invalid argument WARN[0000] Error validating CNI config file /etc/cni/net.d/87-podman-bridge.conflist: [plugin bridge does not support config version "0.4.0" plugin portmap does not support config version "0.4.0" failed to find plugin "firewall" in path [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] plugin tuning does not support config version "0.4.0"] WARN[0000] Default CNI network name podman is unchangeable

any other error but i think it looks good. any suggestions?

[ssbarnea]

I am facing the same error on RHEL-8.1 (updated):

➜  ssbarnea podman --version
podman version 1.6.4-dev
➜  ssbarnea /usr/libexec/podman/conmon --version
conmon version 1.0.0-rc2
commit: 0373794a4d8fa3140406163ef14fe83c821bd339-dirty

podman-1.6.4-1575390589.git47c0dae4.el8.x86_64 : Manage Pods, Containers and Container Images
Repo        : @System
Matched from:
Provide    : podman = 1.6.4-1575390589.git47c0dae4.el8

[ssbarnea]

I updated the rpm building jobs in order to catch this issue, which apparently affects all 3 rpm platforms: see https://review.rdoproject.org/r/#/c/24349/ -- as soon we merge it we will be able to see results on any libpod PR.