Kernel Panic: OpenZFS 2.1.0 Catalina

[sbytnar]

panic(cpu 4 caller 0xffffff802cb5a63c): "free_io_buf: bp(0xffffff8210c1c5f0) - bufstats.bufs_iobufinuse < 0"@/AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/xnu/xnu-6153.141.66/bsd/vfs/vfs_bio.c:4567
Backtrace (CPU 4), Frame : Return Address
0xffffff820deab030 : 0xffffff802c91b54d mach_kernel : _handle_debugger_trap + 0x49d
0xffffff820deab080 : 0xffffff802ca55d85 mach_kernel : _kdp_i386_trap + 0x155
0xffffff820deab0c0 : 0xffffff802ca4790e mach_kernel : _kernel_trap + 0x4ee
0xffffff820deab110 : 0xffffff802c8c1a40 mach_kernel : _return_from_trap + 0xe0
0xffffff820deab130 : 0xffffff802c91ac17 mach_kernel : _DebuggerTrapWithState + 0x17
0xffffff820deab230 : 0xffffff802c91b007 mach_kernel : _panic_trap_to_debugger + 0x227
0xffffff820deab280 : 0xffffff802d0c03bc mach_kernel : _panic + 0x54
0xffffff820deab2f0 : 0xffffff802cb5a63c mach_kernel : _free_io_buf + 0x16c
0xffffff820deab330 : 0xffffff7fb00a043c org.openzfsonosx.zfs : _ldi_vnode_io_intr + 0x2c
0xffffff820deab350 : 0xffffff7fb00a0403 org.openzfsonosx.zfs : _buf_strategy_vnode + 0x223
0xffffff820deab380 : 0xffffff7fb00e04b3 org.openzfsonosx.zfs : _vdev_disk_io_strategy + 0xe3
0xffffff820deab3c0 : 0xffffff7fb0153eca org.openzfsonosx.zfs : _zio_vdev_io_start + 0x1da
0xffffff820deab400 : 0xffffff7fb01500bc org.openzfsonosx.zfs : _zio_nowait + 0x15c
0xffffff820deab430 : 0xffffff7fb00eb854 org.openzfsonosx.zfs : _vdev_mirror_io_start + 0x124
0xffffff820deab480 : 0xffffff7fb0153eca org.openzfsonosx.zfs : _zio_vdev_io_start + 0x1da
0xffffff820deab4c0 : 0xffffff7fb01500bc org.openzfsonosx.zfs : _zio_nowait + 0x15c
0xffffff820deab4f0 : 0xffffff7fb003d6c6 org.openzfsonosx.zfs : _arc_read + 0xc36
0xffffff820deab5a0 : 0xffffff7fb004e59b org.openzfsonosx.zfs : _dbuf_read_impl + 0x27b
0xffffff820deab6b0 : 0xffffff7fb004e028 org.openzfsonosx.zfs : _dbuf_read + 0x338
0xffffff820deab720 : 0xffffff7fb0057134 org.openzfsonosx.zfs : _dmu_buf_hold + 0x44
0xffffff820deab760 : 0xffffff7fb0103365 org.openzfsonosx.zfs : _zap_lockdir + 0x35
0xffffff820deab7b0 : 0xffffff7fb0105207 org.openzfsonosx.zfs : _zap_cursor_retrieve + 0x157
0xffffff820deab840 : 0xffffff7fb0136b3e org.openzfsonosx.zfs : _zfs_readdir + 0x3be
0xffffff820deaba70 : 0xffffff7fb013ccc0 org.openzfsonosx.zfs : _zfs_vnop_readdir + 0xe0
0xffffff820deabaf0 : 0xffffff802cb95d19 mach_kernel : _vnode_readdir64 + 0x3e9
0xffffff820deabba0 : 0xffffff802cb56349 mach_kernel : _getattrlistbulk + 0xa59
0xffffff820deabf40 : 0xffffff802cf83517 mach_kernel : _unix_syscall64 + 0x287
0xffffff820deabfa0 : 0xffffff802c8c2206 mach_kernel : _hndl_unix_scall64 + 0x16
Kernel Extensions in backtrace:
org.openzfsonosx.zfs(2.1)[D8D37025-5993-3A02-8E85-9537CACFBA27]@0xffffff7fb0037000->0xffffff7fb17dcfff
dependency: com.apple.iokit.IOStorageFamily(2.1)[77F6B260-53DF-3DC0-B89F-5205E17FF986]@0xffffff7fad265000

This has been working great, forever, until now... Carbon Copy Cloner was backing up to a disk image.

The pool did not automount on reboot after this panic.
zpool import worked.
zpool status -v shows DEGRADED and permanent errors are in files associated with a disk image that is used to backup.

123GB free on the 3TB 5400RPM disk in question. smartctl status shows no problems. However, it will take several hours to get an extended smartctl test result.

[lundman]

Sounds like it tried to free a buf too many times, and the count of bugs went zero.

It is also curious you are in buf_strategy_vnode() so you have picked to use vnode style over iokit? The default is iokit - out of curiosity, any reason for this?

This should have been a one-time crash, you could try scrub to see if the files come back as OK