NixOS 23.05: crun: kernel 6.1.59 breaks compatibility with crun < 1.9.2

[n-riesco]

Describe the bug

Today, I upgraded my NixOS 23.05 and the kernel 6.1.59 was installed.

After completing the upgrade, podman started throwing the following error when I run some of my containers:

Error: OCI runtime error: crun: chmod `run/shm`: Operation not supported

According to this issue in Debian, this error was caused by the kernel upgrade breaking compatibility with crun < 1.9.2.

Steps To Reproduce

1. See the comment here.

Expected behavior

No runtime error.

Additional context

# podman info
host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /nix/store/0yvag7pcqwcgq8cjhrfqsp97njp328wm-conmon-2.1.7/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 96.84
    systemPercent: 0.9
    userPercent: 2.26
  cpus: 4
  databaseBackend: boltdb
  distribution:
    codename: stoat
    distribution: nixos
    version: "23.05"
  eventLogger: journald
  hostname: lexx
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.1.59
  linkmode: dynamic
  logDriver: journald
  memFree: 523460608
  memTotal: 15688220672
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: Unknown
    path: /nix/store/71xi7lr64sagp70wwxcm95w00jdynjdz-crun-1.8.4/bin/crun
    version: |-
      crun version 1.8.4
      commit: 1.8.4
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /nix/store/bv4przd06r36f0dw00xz6v64hdzbbzmm-slirp4netns-1.2.0/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 17250304000
  swapTotal: 17255321600
  uptime: 4h 16m 0.00s (Approximately 0.17 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 13
    paused: 0
    running: 0
    stopped: 13
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 1950855696384
  graphRootUsed: 1206369812480
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 230
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.5.0
  Built: 315532800
  BuiltTime: Tue Jan  1 00:00:00 1980
  GitCommit: ""
  GoVersion: go1.20.8
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.0

Notify maintainers

@adisbladis
@saschagrunert
@vdemeester
@zowoq

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.59, NixOS, 23.05 (Stoat), 23.05.4738.41de143fda10`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.13.6`
 - channels(root): `"nixos-23.05"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

[n-riesco]

The issue is fixed in podman v4.7.2 from nixos-unstable (as it's linked to crun v1.11.1).

---

Tested using the following overlay:

  nixpkgs.overlays = let
    # Ensure <nixos-unstable> is defined:
    #   sudo nix-channel --add https://nixos.org/channels/nixos-unstable nixos-unstable
    # And to update:
    #   sudo nix-channel --update
    nixos-unstable = import <nixos-unstable> {};
  in [
    # Install podman from unstable.
    # See:
    #   https://github.com/NixOS/nixpkgs/issues/266302
    (final: prev: {
      inherit (nixos-unstable.pkgs) podman podman-compose podman-desktop podman-tui;
    })
  ];

[ashuramaruzxc]

The issue is fixed in podman v4.7.2 from nixos-unstable (as it's linked to crun v1.11.1).

No, it wasn't i still get errors from crun. More than that, running podman as root seems to be completely broken on nixos unstable for now.

# podman run docker.io/library/alpine:latest

output: Error: OCI runtime error: crun: {"msg":"read from sync socket","level": "error", "time": "2023-11-10T11:46:47.000226Z"}"

[saschagrunert]

Hm, it seems to work everything as intended on nixpkgs-unstable, so it feels that we just have to upgrade podman and its dependencies on nixos-unstable, right?