Podman 2.1.1 libpod/events endpoint returns status code 500

[j2mcv]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

The endpoint always returns status code 500

Steps to reproduce the issue:

1. Install podman 2.1.1

2. curl -s --unix-socket /run/podman/podman.sock http://d/v1.12/libpod/events

Describe the results you received:

{"cause":"failed to read message field: bad message","message":"failed to read message field: bad message","response":500}

Describe the results you expected:

a string of json data describing an event

Additional information you deem important (e.g. issue happens only occasionally):

cockpit-podman returns that the service is not active, as reported here:
cockpit-project/cockpit-podman#546

Output of podman version:

Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.15.2
Built:        Mon Oct  5 01:25:24 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.21-3.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 0f53fb68333bdead5fe4dc5175703e22cf9882ab'
  cpus: 4
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: piticli.lan
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.8.15-301.fc33.x86_64
  linkmode: dynamic
  memFree: 4887814144
  memTotal: 7972184064
  ociRuntime:
    name: crun
    package: crun-0.15-5.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.15
      commit: 56ca95e61639510c7dbd39ff512f80f626404969
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootless: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 4155502592
  swapTotal: 4155502592
  uptime: 59h 3m 43.8s (Approximately 2.46 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 0
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1601853924
  BuiltTime: Mon Oct  5 01:25:24 2020
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 2.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.1.1-10.fc33.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes, this package is the last version available on fedora 33.

Additional environment details (AWS, VirtualBox, physical, etc.):

physical machine.

[Luap99]

This is not an api problem. If you run podman events you should get the same error.

The error is unfortunately not very informative. Looking at the code there is a problem while reading the journal. Does journalctl --verify reports a corruption?

[mheon]

I had this problem before, and it turned out to be a spurious journal entry with the word "podman" in it that did not properly decode as JSON.

[j2mcv]

This is not an api problem. If you run podman events you should get the same error.

The error is unfortunately not very informative. Looking at the code there is a problem while reading the journal. Does journalctl --verify reports a corruption?

I had 2 corrupted files in the journal, after removing them I do not get more error responses.

Certainly the message coul be more informative.

Cheers.

[j0057]

I have the same issue, some corrupted journal files. For me, this works as a work-around: journalctl --follow --since=now --output=json PODMAN_TYPE=container.

[Luap99]

Could you try podman events --since 1s if podman events still fails?

[j0057]

With --since 1s it works with the corrupted files still present.

[Luap99]

Thanks for the confirmation. PR #8164 will fix this problem.