Adjusting zfs_vdev_aggregation_limit caused lockup, corrupt pool #520

akorn · 2012-01-08T00:00:25Z

Hi,

as part of my experiment to test the robustness of zfsonlinux, I raised zfs_vdev_aggregation_limit to 1048576 (from the default 131072) by echoing that value into /sys/module/zfs/parameters/zfs_vdev_aggregation_limit.

A few seconds later the box locked up and likely kernel panicked, because it rebooted.

After the reboot, my pool was in the following state:

pool: tank
state: UNAVAIL
status: One or more devices could not be used because the label is missing
or invalid. There are insufficient replicas for the pool to continue
functioning.
action: Destroy and re-create the pool from
a backup source.
see: http://zfsonlinux.org/msg/ZFS-8000-5E
scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    tank        UNAVAIL      0     0     0  insufficient replicas
      raidz2-0  UNAVAIL      0     0     0  insufficient replicas
        sdc     UNAVAIL      0     0     0  corrupted data
        sdd     UNAVAIL      0     0     0  corrupted data
        sde     FAULTED      0     0     0  corrupted data
        sdf     FAULTED      0     0     0  corrupted data
        sdg     FAULTED      0     0     0  corrupted data
    logs
      zfs_zil   ONLINE       0     0     0

The write cache on the drives had been disabled.

I suspect there is no way to recover from this state (which is "fine", it wasn't a live system).

Unfortunately, I wasn't able to capture the kernel message, if any.

The last interesting bit from the kernel log was logged about an hour earlier:

Jan 7 22:12:05 jayhawk kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x20)
Jan 7 22:12:05 jayhawk kernel: cache: kmalloc-2048, object size: 2048, buffer size: 2048, default order: 3, min order: 0
Jan 7 22:12:05 jayhawk kernel: node 0: slabs: 583, objs: 9328, free: 0
Jan 7 22:12:05 jayhawk kernel: arc_reclaim: page allocation failure: order:0, mode:0x4020
Jan 7 22:12:05 jayhawk kernel: Pid: 1906, comm: arc_reclaim Tainted: P 3.1.6-vs2.3.2.5-newjayhawk #4
Jan 7 22:12:05 jayhawk kernel: Call Trace:
Jan 7 22:12:05 jayhawk kernel: [] warn_alloc_failed+0xe6/0x140
Jan 7 22:12:05 jayhawk kernel: [] __alloc_pages_nodemask+0x5df/0x800
Jan 7 22:12:05 jayhawk kernel: [] alloc_pages_current+0x99/0x100
Jan 7 22:12:05 jayhawk kernel: [] new_slab+0x2a9/0x2c0
Jan 7 22:12:05 jayhawk kernel: [] __slab_alloc+0x2ce/0x3fe
Jan 7 22:12:05 jayhawk kernel: [] ? __netdev_alloc_skb+0x1f/0x40
Jan 7 22:12:05 jayhawk kernel: [] ? __alloc_skb+0x7e/0x220
Jan 7 22:12:05 jayhawk kernel: [] __kmalloc_node_track_caller+0x84/0x1a0
Jan 7 22:12:05 jayhawk kernel: [] ? __alloc_skb+0x4a/0x220
Jan 7 22:12:05 jayhawk kernel: [] ? __netdev_alloc_skb+0x1f/0x40
Jan 7 22:12:05 jayhawk kernel: [] __alloc_skb+0x7e/0x220
Jan 7 22:12:05 jayhawk kernel: [] __netdev_alloc_skb+0x1f/0x40
Jan 7 22:12:05 jayhawk kernel: [] e1000_alloc_rx_buffers+0x1bf/0x2c0 [e1000e]
Jan 7 22:12:05 jayhawk kernel: [] ? napi_gro_receive+0xad/0xc0
Jan 7 22:12:05 jayhawk kernel: [] e1000_clean_rx_irq+0x329/0x420 [e1000e]
Jan 7 22:12:05 jayhawk kernel: [] e1000_clean+0xa7/0x2a0 [e1000e]
Jan 7 22:12:05 jayhawk kernel: [] net_rx_action+0x12c/0x270
Jan 7 22:12:05 jayhawk kernel: [] __do_softirq+0xa0/0x200
Jan 7 22:12:05 jayhawk kernel: [] ? _raw_spin_lock+0x9/0x10
Jan 7 22:12:05 jayhawk kernel: [] ? handle_irq_event+0x54/0x70
Jan 7 22:12:05 jayhawk kernel: [] call_softirq+0x1c/0x30
Jan 7 22:12:05 jayhawk kernel: [] do_softirq+0x65/0xa0
Jan 7 22:12:05 jayhawk kernel: [] irq_exit+0x86/0xa0
Jan 7 22:12:05 jayhawk kernel: [] do_IRQ+0x5e/0xd0
Jan 7 22:12:05 jayhawk kernel: [] common_interrupt+0x6e/0x6e
Jan 7 22:12:05 jayhawk kernel: [] ? l2arc_read_done+0x220/0x220 [zfs]
Jan 7 22:12:05 jayhawk kernel: [] ? buf_dest+0x10/0x90 [zfs]
Jan 7 22:12:05 jayhawk kernel: [] spl_slab_reclaim+0x1a9/0x270 [spl]
Jan 7 22:12:05 jayhawk kernel: [] spl_kmem_cache_reap_now+0x47/0x60 [spl]
Jan 7 22:12:05 jayhawk kernel: [] __spl_kmem_cache_generic_shrinker.isra.10+0x64/0xa0 [spl]
Jan 7 22:12:05 jayhawk kernel: [] spl_kmem_reap+0x20/0x30 [spl]
Jan 7 22:12:05 jayhawk kernel: [] arc_kmem_reap_now+0x8e/0x110 [zfs]
Jan 7 22:12:05 jayhawk kernel: [] arc_reclaim_thread+0x6f/0x150 [zfs]
Jan 7 22:12:05 jayhawk kernel: [] ? arc_shrinker_func+0xd0/0xd0 [zfs]
Jan 7 22:12:05 jayhawk kernel: [] thread_generic_wrapper+0x73/0x90 [spl]
Jan 7 22:12:05 jayhawk kernel: [] ? __thread_create+0x160/0x160 [spl]
Jan 7 22:12:05 jayhawk kernel: [] kthread+0x87/0x90
Jan 7 22:12:05 jayhawk kernel: [] kernel_thread_helper+0x4/0x10
Jan 7 22:12:05 jayhawk kernel: [] ? flush_kthread_worker+0xa0/0xa0
Jan 7 22:12:05 jayhawk kernel: [] ? gs_change+0x13/0x13
Jan 7 22:12:05 jayhawk kernel: Mem-Info:
Jan 7 22:12:05 jayhawk kernel: Node 0 DMA per-cpu:
Jan 7 22:12:05 jayhawk kernel: CPU 0: hi: 0, btch: 1 usd: 0
Jan 7 22:12:05 jayhawk kernel: CPU 1: hi: 0, btch: 1 usd: 0
Jan 7 22:12:05 jayhawk kernel: CPU 2: hi: 0, btch: 1 usd: 0
Jan 7 22:12:05 jayhawk kernel: CPU 3: hi: 0, btch: 1 usd: 0
Jan 7 22:12:05 jayhawk kernel: CPU 4: hi: 0, btch: 1 usd: 0
Jan 7 22:12:05 jayhawk kernel: CPU 5: hi: 0, btch: 1 usd: 0
Jan 7 22:12:05 jayhawk kernel: Node 0 DMA32 per-cpu:
Jan 7 22:12:05 jayhawk kernel: CPU 0: hi: 186, btch: 31 usd: 0
Jan 7 22:12:05 jayhawk kernel: CPU 1: hi: 186, btch: 31 usd: 0
Jan 7 22:12:05 jayhawk kernel: CPU 2: hi: 186, btch: 31 usd: 0
Jan 7 22:12:05 jayhawk kernel: CPU 3: hi: 186, btch: 31 usd: 0
Jan 7 22:12:05 jayhawk kernel: CPU 4: hi: 186, btch: 31 usd: 163
Jan 7 22:12:05 jayhawk kernel: CPU 5: hi: 186, btch: 31 usd: 0
Jan 7 22:12:05 jayhawk kernel: Node 0 Normal per-cpu:
Jan 7 22:12:05 jayhawk kernel: CPU 0: hi: 186, btch: 31 usd: 0
Jan 7 22:12:05 jayhawk kernel: CPU 1: hi: 186, btch: 31 usd: 107
Jan 7 22:12:05 jayhawk kernel: CPU 2: hi: 186, btch: 31 usd: 0
Jan 7 22:12:05 jayhawk kernel: CPU 3: hi: 186, btch: 31 usd: 90
Jan 7 22:12:05 jayhawk kernel: CPU 4: hi: 186, btch: 31 usd: 162
Jan 7 22:12:05 jayhawk kernel: CPU 5: hi: 186, btch: 31 usd: 0
Jan 7 22:12:05 jayhawk kernel: active_anon:170738 inactive_anon:23028 isolated_anon:0
Jan 7 22:12:05 jayhawk kernel: active_file:107137 inactive_file:192920 isolated_file:0
Jan 7 22:12:05 jayhawk kernel: unevictable:516 dirty:5 writeback:0 unstable:0
Jan 7 22:12:05 jayhawk kernel: free:7978 slab_reclaimable:544352 slab_unreclaimable:427758
Jan 7 22:12:05 jayhawk kernel: mapped:1827 shmem:146 pagetables:1360 bounce:0
Jan 7 22:12:05 jayhawk kernel: Node 0 DMA free:8084kB min:60kB low:72kB high:88kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15628kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Jan 7 22:12:05 jayhawk kernel: lowmem_reserve[]: 0 3502 16102 16102
Jan 7 22:12:05 jayhawk kernel: Node 0 DMA32 free:9220kB min:14684kB low:18352kB high:22024kB active_anon:101156kB inactive_anon:24076kB active_file:36596kB inactive_file:269076kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3586880kB mlocked:0kB dirty:4kB writeback:0kB mapped:28kB shmem:0kB slab_reclaimable:594536kB slab_unreclaimable:306060kB kernel_stack:72kB pagetables:144kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:16 all_unreclaimable? no
Jan 7 22:12:05 jayhawk kernel: lowmem_reserve[]: 0 0 12600 12600
Jan 7 22:12:05 jayhawk kernel: Node 0 Normal free:16592kB min:52832kB low:66040kB high:79248kB active_anon:581992kB inactive_anon:68036kB active_file:391952kB inactive_file:500468kB unevictable:2064kB isolated(anon):0kB isolated(file):0kB present:12902400kB mlocked:2064kB dirty:16kB writeback:0kB mapped:7280kB shmem:584kB slab_reclaimable:1582676kB slab_unreclaimable:1404972kB kernel_stack:4224kB pagetables:5296kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jan 7 22:12:05 jayhawk kernel: lowmem_reserve[]: 0 0 0 0
Jan 7 22:12:05 jayhawk kernel: Node 0 DMA: 1_4kB 0_8kB 1_16kB 0_32kB 0_64kB 1_128kB 1_256kB 1_512kB 1_1024kB 1_2048kB 1_4096kB = 8084kB
Jan 7 22:12:05 jayhawk kernel: Node 0 DMA32: 568_4kB 252_8kB 171_16kB 93_32kB 7_64kB 0_128kB 0_256kB 0_512kB 0_1024kB 0_2048kB 0_4096kB = 10448kB
Jan 7 22:12:05 jayhawk kernel: Node 0 Normal: 3007_4kB 476_8kB 122_16kB 14_32kB 0_64kB 0_128kB 0_256kB 0_512kB 0_1024kB 0_2048kB 0*4096kB = 18236kB
Jan 7 22:12:05 jayhawk kernel: 303055 total pagecache pages
Jan 7 22:12:05 jayhawk kernel: 3506 pages in swap cache
Jan 7 22:12:05 jayhawk kernel: Swap cache stats: add 1677496, delete 1673990, find 677188/741654
Jan 7 22:12:05 jayhawk kernel: Free swap = 3849660kB
Jan 7 22:12:05 jayhawk kernel: Total swap = 4194300kB
Jan 7 22:12:05 jayhawk kernel: 4194288 pages RAM
Jan 7 22:12:05 jayhawk kernel: 101818 pages reserved
Jan 7 22:12:05 jayhawk kernel: 291975 pages shared
Jan 7 22:12:05 jayhawk kernel: 2368120 pages non-shared
Jan 7 22:12:05 jayhawk kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x20)
Jan 7 22:12:05 jayhawk kernel: cache: kmalloc-2048, object size: 2048, buffer size: 2048, default order: 3, min order: 0
Jan 7 22:12:05 jayhawk kernel: node 0: slabs: 583, objs: 9328, free: 0

The text was updated successfully, but these errors were encountered:

akorn · 2012-01-08T00:10:11Z

Sorry, no, that's not actually the problem.

The problem seems to be that two drives had been hot-plugged, which used to be sdh and sdi, but are now (after the reboot) sdc and sdd, taking the place of the drives that were in the pool.

akorn · 2012-01-08T00:18:50Z

Yes, that's what it was. Exporting and importing the pool seems to have worked.

So the only issue seems to be that adjusting zfs_vdev_aggregation_limit caused a kernel panic.

behlendorf · 2012-01-13T00:26:48Z

OK, thanks for clarifying that because there's no way that should have corrupted you pool. The crash on the other hand I can entirely believe and should be pretty easy to recreate when someone get's some time to look at it.

Stoatwblr · 2012-01-14T17:51:16Z

FWIW, setting this value to anything larger than default results in spl panicing at mount time.

behlendorf · 2012-01-17T20:43:15Z

Hmm, I take it back. Increasing the zfs_vdev_aggregation_limit may have damaged your pool, in particular the space maps which are stored on disk. I've seen this sort of damage once before. Can you try setting the zfs_recover=1 module option and scrubbing your pool. You probably also want to make sure you have an up to date backup, you may need to recreate your pool to fix this.

[ 4463.017339] zfs: allocating allocated segment(offset=2516976500736
size=196608)
[ 4463.036249] SPLError: 6266:0:(spl-err.c:48:vpanic()) SPL PANIC
[ 4463.045747] SPL: Showing stack for process 6266
[ 4463.054980] Pid: 6266, comm: z_fr_iss/99 Tainted: P         C O
3.2.0-9-generic #16-Ubuntu
[ 4463.073504] Call Trace:
[ 4463.082469]  [] spl_debug_dumpstack+0x27/0x40 [spl]
[ 4463.091436]  [] spl_debug_bug+0x82/0xe0 [spl]
[ 4463.100277]  [] vpanic+0x65/0x80 [spl]
[ 4463.108956]  [] ? kmem_alloc_debug+0xeb/0x130 [spl]
[ 4463.117570]  [] ? kmem_alloc_debug+0xeb/0x130 [spl]
[ 4463.125894]  [] vcmn_err+0x3f/0x50 [spl]
[ 4463.134091]  [] zfs_panic_recover+0x52/0x60 [zfs]
[ 4463.142180]  [] space_map_add+0x254/0x3c0 [zfs]
[ 4463.150091]  [] metaslab_free_dva+0x123/0x1d0 [zfs]
[ 4463.157882]  [] metaslab_free+0x8c/0xc0 [zfs]
[ 4463.165492]  [] zio_dva_free+0x1c/0x30 [zfs]
[ 4463.172951]  [] zio_execute+0x9f/0xf0 [zfs]
[ 4463.180227]  [] taskq_thread+0x1e3/0x420 [spl]
[ 4463.187383]  [] ? try_to_wake_up+0x200/0x200
[ 4463.194394]  [] ? task_alloc+0x160/0x160 [spl]
[ 4463.201240]  [] kthread+0x8c/0xa0
[ 4463.207933]  [] kernel_thread_helper+0x4/0x10
[ 4463.214608]  [] ? flush_kthread_worker+0xa0/0xa0
[ 4463.221166]  [] ? gs_change+0x13/0x13
[ 4463.227648] SPL: Dumping log to /tmp/spl-log.1326623409.6266

akorn · 2012-01-17T20:59:21Z

How dangerous is setting zfs_recover=1? Meanwhile, this box went live-ish, and there have been no readily apparent problems with the pool so far.

behlendorf · 2012-01-17T21:37:42Z

It's actually pretty safe. All it does is convert what zfs considers a fatal error in to something it tries to deal with. In the stack posted above it would cause the zfs_panic_recover() function to simply print a warning and try to continue, rather than panicing the system which is what does now. The safest course of action would be.

Scrub your pool with zfs_recover=0. Hopefully, this returns no errors/panics and we conclude the space maps have subsequently updated and everything is fine now.
If the scrub fails, set zfs_recover=1 and scrub your pool again to try and safely force the space maps to be updated.

Stoatwblr · 2012-01-17T23:42:42Z

On 17/01/12 21:37, Brian Behlendorf wrote:

It's actually pretty safe. All it does is convert what zfs considers a fatal error in to something it tries to deal with.

Everything is back and the scrub went cleanly.

Still, quite worrying when it happened.

I have to say I'm impressed. Despite all the abuse I've thrown at the
system (including dropping 4 out of 9 drives in a raidz2 pool when a
controller went south) it's come back intact every time.

behlendorf · 2012-01-18T00:05:44Z

That's good to hear! Well if you aren't observing any troubles and the scrub came back clean your probably OK.

behlendorf · 2012-01-23T20:46:44Z

I've taken a careful look at this case and for the moment increasing zfs_vdev_aggregation_limit beyond 128k will absolutely result in corruption. Basically what happens is when an I/O tries to get aggregated beyond 128k is that the data past this point just gets dropped, however zfs is told the I/O completed. There are assertions in the code to catch this case, if you were running with --enable-debug it would have asserted immediately before damaging the pool.

For my own reasons I'm looking at how this limit can be increased to improve performance. Until then however I may land a patch to preventing anyone else from increasing this value.

akorn · 2012-01-23T21:43:24Z

FWIW, the scrub found everything in order on my pool. Maybe the kernel crashed before corruption could occur? (Just musing.)

behlendorf · 2012-01-25T17:20:54Z

Yes, that's a likely outcome since we'll overrun the buffer in memory.

behlendorf · 2012-06-13T19:11:10Z

This is more of a documentation issue that a bug, simply don't increase your zfs_vdev_aggregation_limit beyond 128K. I've leaving the issue open however until we get a chance to add a patch which prevents you from doing this.

Prevent users from setting the zfs_vdev_aggregation_limit tuning larger than SPA_MAXBLOCKSIZE. Signed-off-by: Brian Behlendorf <[email protected]> Closes openzfs#520

The below excerpt of a backtrace is from a ztest failure when running ZoL's ztest. /openzfs#453 0x00007f03c8060b35 in vdev_queue_io_to_issue (vq=vq@entry=0x99f8a8) at ../../module/zfs/vdev_queue.c:706 /openzfs#454 0x00007f03c806106e in vdev_queue_io (zio=zio@entry=0x7f0350003de0) at ../../module/zfs/vdev_queue.c:747 /openzfs#455 0x00007f03c80818c1 in zio_vdev_io_start (zio=0x7f0350003de0) at ../../module/zfs/zio.c:2659 /openzfs#456 0x00007f03c807f243 in __zio_execute (zio=0x7f0350003de0) at ../../module/zfs/zio.c:1399 /openzfs#457 zio_nowait (zio=0x7f0350003de0) at ../../module/zfs/zio.c:1456 /openzfs#458 0x00007f03c805f71b in vdev_mirror_io_start (zio=0x7f0350003a10) at ../../module/zfs/vdev_mirror.c:374 /openzfs#459 0x00007f03c807f243 in __zio_execute (zio=0x7f0350003a10) at ../../module/zfs/zio.c:1399 /openzfs#460 zio_nowait (zio=0x7f0350003a10) at ../../module/zfs/zio.c:1456 /openzfs#461 0x00007f03c806464c in vdev_raidz_io_start (zio=0x7f0350003380) at ../../module/zfs/vdev_raidz.c:1607 /openzfs#462 0x00007f03c807f243 in __zio_execute (zio=0x7f0350003380) at ../../module/zfs/zio.c:1399 /openzfs#463 zio_nowait (zio=0x7f0350003380) at ../../module/zfs/zio.c:1456 /openzfs#464 0x00007f03c805f71b in vdev_mirror_io_start (zio=0x7f0350002fb0) at ../../module/zfs/vdev_mirror.c:374 /openzfs#465 0x00007f03c807f243 in __zio_execute (zio=0x7f0350002fb0) at ../../module/zfs/zio.c:1399 /openzfs#466 zio_nowait (zio=0x7f0350002fb0) at ../../module/zfs/zio.c:1456 /openzfs#467 0x00007f03c805ed43 in vdev_mirror_io_done (zio=0x7f033957ebf0) at ../../module/zfs/vdev_mirror.c:499 /openzfs#468 0x00007f03c807a0c0 in zio_vdev_io_done (zio=0x7f033957ebf0) at ../../module/zfs/zio.c:2707 /openzfs#469 0x00007f03c808285b in __zio_execute (zio=0x7f033957ebf0) at ../../module/zfs/zio.c:1399 /openzfs#470 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f0390001330, pio=0x7f033957ebf0) at ../../module/zfs/zio.c:547 /openzfs#471 zio_done (zio=0x7f0390001330) at ../../module/zfs/zio.c:3278 /openzfs#472 0x00007f03c808285b in __zio_execute (zio=0x7f0390001330) at ../../module/zfs/zio.c:1399 /openzfs#473 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f03b4013a00, pio=0x7f0390001330) at ../../module/zfs/zio.c:547 /openzfs#474 zio_done (zio=0x7f03b4013a00) at ../../module/zfs/zio.c:3278 /openzfs#475 0x00007f03c808285b in __zio_execute (zio=0x7f03b4013a00) at ../../module/zfs/zio.c:1399 /openzfs#476 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f03b4014210, pio=0x7f03b4013a00) at ../../module/zfs/zio.c:547 /openzfs#477 zio_done (zio=0x7f03b4014210) at ../../module/zfs/zio.c:3278 /openzfs#478 0x00007f03c808285b in __zio_execute (zio=0x7f03b4014210) at ../../module/zfs/zio.c:1399 /openzfs#479 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f03b4014620, pio=0x7f03b4014210) at ../../module/zfs/zio.c:547 /openzfs#480 zio_done (zio=0x7f03b4014620) at ../../module/zfs/zio.c:3278 /openzfs#481 0x00007f03c807a6d3 in __zio_execute (zio=0x7f03b4014620) at ../../module/zfs/zio.c:1399 /openzfs#482 zio_execute (zio=zio@entry=0x7f03b4014620) at ../../module/zfs/zio.c:1337 /openzfs#483 0x00007f03c8060b35 in vdev_queue_io_to_issue (vq=vq@entry=0x99f8a8) at ../../module/zfs/vdev_queue.c:706 /openzfs#484 0x00007f03c806106e in vdev_queue_io (zio=zio@entry=0x7f0350002be0) at ../../module/zfs/vdev_queue.c:747 /openzfs#485 0x00007f03c80818c1 in zio_vdev_io_start (zio=0x7f0350002be0) at ../../module/zfs/zio.c:2659 /openzfs#486 0x00007f03c807f243 in __zio_execute (zio=0x7f0350002be0) at ../../module/zfs/zio.c:1399 /openzfs#487 zio_nowait (zio=0x7f0350002be0) at ../../module/zfs/zio.c:1456 /openzfs#488 0x00007f03c805f71b in vdev_mirror_io_start (zio=0x7f0350002810) at ../../module/zfs/vdev_mirror.c:374 /openzfs#489 0x00007f03c807f243 in __zio_execute (zio=0x7f0350002810) at ../../module/zfs/zio.c:1399 /openzfs#490 zio_nowait (zio=0x7f0350002810) at ../../module/zfs/zio.c:1456 /openzfs#491 0x00007f03c8064593 in vdev_raidz_io_start (zio=0x7f0350001270) at ../../module/zfs/vdev_raidz.c:1591 /openzfs#492 0x00007f03c807f243 in __zio_execute (zio=0x7f0350001270) at ../../module/zfs/zio.c:1399 /openzfs#493 zio_nowait (zio=0x7f0350001270) at ../../module/zfs/zio.c:1456 /openzfs#494 0x00007f03c805f71b in vdev_mirror_io_start (zio=0x7f0350001e60) at ../../module/zfs/vdev_mirror.c:374 /openzfs#495 0x00007f03c807f243 in __zio_execute (zio=0x7f0350001e60) at ../../module/zfs/zio.c:1399 /openzfs#496 zio_nowait (zio=0x7f0350001e60) at ../../module/zfs/zio.c:1456 /openzfs#497 0x00007f03c805ed43 in vdev_mirror_io_done (zio=0x7f033a0c39c0) at ../../module/zfs/vdev_mirror.c:499 /openzfs#498 0x00007f03c807a0c0 in zio_vdev_io_done (zio=0x7f033a0c39c0) at ../../module/zfs/zio.c:2707 /openzfs#499 0x00007f03c808285b in __zio_execute (zio=0x7f033a0c39c0) at ../../module/zfs/zio.c:1399 /openzfs#500 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f03a8003c00, pio=0x7f033a0c39c0) at ../../module/zfs/zio.c:547 /openzfs#501 zio_done (zio=0x7f03a8003c00) at ../../module/zfs/zio.c:3278 /openzfs#502 0x00007f03c808285b in __zio_execute (zio=0x7f03a8003c00) at ../../module/zfs/zio.c:1399 /openzfs#503 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f038800c400, pio=0x7f03a8003c00) at ../../module/zfs/zio.c:547 /openzfs#504 zio_done (zio=0x7f038800c400) at ../../module/zfs/zio.c:3278 /openzfs#505 0x00007f03c808285b in __zio_execute (zio=0x7f038800c400) at ../../module/zfs/zio.c:1399 /openzfs#506 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f038800da00, pio=0x7f038800c400) at ../../module/zfs/zio.c:547 /openzfs#507 zio_done (zio=0x7f038800da00) at ../../module/zfs/zio.c:3278 /openzfs#508 0x00007f03c808285b in __zio_execute (zio=0x7f038800da00) at ../../module/zfs/zio.c:1399 /openzfs#509 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f038800fd80, pio=0x7f038800da00) at ../../module/zfs/zio.c:547 /openzfs#510 zio_done (zio=0x7f038800fd80) at ../../module/zfs/zio.c:3278 /openzfs#511 0x00007f03c807a6d3 in __zio_execute (zio=0x7f038800fd80) at ../../module/zfs/zio.c:1399 /openzfs#512 zio_execute (zio=zio@entry=0x7f038800fd80) at ../../module/zfs/zio.c:1337 /openzfs#513 0x00007f03c8060b35 in vdev_queue_io_to_issue (vq=vq@entry=0x99f8a8) at ../../module/zfs/vdev_queue.c:706 /openzfs#514 0x00007f03c806119d in vdev_queue_io_done (zio=zio@entry=0x7f03a0010950) at ../../module/zfs/vdev_queue.c:775 /openzfs#515 0x00007f03c807a0e8 in zio_vdev_io_done (zio=0x7f03a0010950) at ../../module/zfs/zio.c:2686 /openzfs#516 0x00007f03c807a6d3 in __zio_execute (zio=0x7f03a0010950) at ../../module/zfs/zio.c:1399 /openzfs#517 zio_execute (zio=0x7f03a0010950) at ../../module/zfs/zio.c:1337 /openzfs#518 0x00007f03c7fcd0c4 in taskq_thread (arg=0x966d50) at ../../lib/libzpool/taskq.c:215 /openzfs#519 0x00007f03c7fc7937 in zk_thread_helper (arg=0x967e90) at ../../lib/libzpool/kernel.c:135 /openzfs#520 0x00007f03c78890a3 in start_thread (arg=0x7f03c2703700) at pthread_create.c:309 /openzfs#521 0x00007f03c75c50fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 The backtrace was an infinite loop of `vdev_queue_io_to_issue()` invoking `zio_execute()` until it overran the stack. vdev_queue_io_to_issue() will ony invoke `zio_execute()` on raidz vdevs when aggregation I/Os are generated to improve aggregation continuity. These I/Os do not trigger any writes. However, it appears that they can be generated in such a way that they recurse infinitely upon return to `vdev_queue_io_to_issue()`. As a consequence, we see the number of parents by 1 each time the recursion returns to `vdev_raidz_io_start()`. Signed-off-by: Richard Yao <[email protected]>

The below excerpt of a backtrace is from a ztest failure when running ZoL's ztest. /openzfs#453 0x00007f03c8060b35 in vdev_queue_io_to_issue (vq=vq@entry=0x99f8a8) at ../../module/zfs/vdev_queue.c:706 /openzfs#454 0x00007f03c806106e in vdev_queue_io (zio=zio@entry=0x7f0350003de0) at ../../module/zfs/vdev_queue.c:747 /openzfs#455 0x00007f03c80818c1 in zio_vdev_io_start (zio=0x7f0350003de0) at ../../module/zfs/zio.c:2659 /openzfs#456 0x00007f03c807f243 in __zio_execute (zio=0x7f0350003de0) at ../../module/zfs/zio.c:1399 /openzfs#457 zio_nowait (zio=0x7f0350003de0) at ../../module/zfs/zio.c:1456 /openzfs#458 0x00007f03c805f71b in vdev_mirror_io_start (zio=0x7f0350003a10) at ../../module/zfs/vdev_mirror.c:374 /openzfs#459 0x00007f03c807f243 in __zio_execute (zio=0x7f0350003a10) at ../../module/zfs/zio.c:1399 /openzfs#460 zio_nowait (zio=0x7f0350003a10) at ../../module/zfs/zio.c:1456 /openzfs#461 0x00007f03c806464c in vdev_raidz_io_start (zio=0x7f0350003380) at ../../module/zfs/vdev_raidz.c:1607 /openzfs#462 0x00007f03c807f243 in __zio_execute (zio=0x7f0350003380) at ../../module/zfs/zio.c:1399 /openzfs#463 zio_nowait (zio=0x7f0350003380) at ../../module/zfs/zio.c:1456 /openzfs#464 0x00007f03c805f71b in vdev_mirror_io_start (zio=0x7f0350002fb0) at ../../module/zfs/vdev_mirror.c:374 /openzfs#465 0x00007f03c807f243 in __zio_execute (zio=0x7f0350002fb0) at ../../module/zfs/zio.c:1399 /openzfs#466 zio_nowait (zio=0x7f0350002fb0) at ../../module/zfs/zio.c:1456 /openzfs#467 0x00007f03c805ed43 in vdev_mirror_io_done (zio=0x7f033957ebf0) at ../../module/zfs/vdev_mirror.c:499 /openzfs#468 0x00007f03c807a0c0 in zio_vdev_io_done (zio=0x7f033957ebf0) at ../../module/zfs/zio.c:2707 /openzfs#469 0x00007f03c808285b in __zio_execute (zio=0x7f033957ebf0) at ../../module/zfs/zio.c:1399 /openzfs#470 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f0390001330, pio=0x7f033957ebf0) at ../../module/zfs/zio.c:547 /openzfs#471 zio_done (zio=0x7f0390001330) at ../../module/zfs/zio.c:3278 /openzfs#472 0x00007f03c808285b in __zio_execute (zio=0x7f0390001330) at ../../module/zfs/zio.c:1399 /openzfs#473 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f03b4013a00, pio=0x7f0390001330) at ../../module/zfs/zio.c:547 /openzfs#474 zio_done (zio=0x7f03b4013a00) at ../../module/zfs/zio.c:3278 /openzfs#475 0x00007f03c808285b in __zio_execute (zio=0x7f03b4013a00) at ../../module/zfs/zio.c:1399 /openzfs#476 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f03b4014210, pio=0x7f03b4013a00) at ../../module/zfs/zio.c:547 /openzfs#477 zio_done (zio=0x7f03b4014210) at ../../module/zfs/zio.c:3278 /openzfs#478 0x00007f03c808285b in __zio_execute (zio=0x7f03b4014210) at ../../module/zfs/zio.c:1399 /openzfs#479 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f03b4014620, pio=0x7f03b4014210) at ../../module/zfs/zio.c:547 /openzfs#480 zio_done (zio=0x7f03b4014620) at ../../module/zfs/zio.c:3278 /openzfs#481 0x00007f03c807a6d3 in __zio_execute (zio=0x7f03b4014620) at ../../module/zfs/zio.c:1399 /openzfs#482 zio_execute (zio=zio@entry=0x7f03b4014620) at ../../module/zfs/zio.c:1337 /openzfs#483 0x00007f03c8060b35 in vdev_queue_io_to_issue (vq=vq@entry=0x99f8a8) at ../../module/zfs/vdev_queue.c:706 /openzfs#484 0x00007f03c806106e in vdev_queue_io (zio=zio@entry=0x7f0350002be0) at ../../module/zfs/vdev_queue.c:747 /openzfs#485 0x00007f03c80818c1 in zio_vdev_io_start (zio=0x7f0350002be0) at ../../module/zfs/zio.c:2659 /openzfs#486 0x00007f03c807f243 in __zio_execute (zio=0x7f0350002be0) at ../../module/zfs/zio.c:1399 /openzfs#487 zio_nowait (zio=0x7f0350002be0) at ../../module/zfs/zio.c:1456 /openzfs#488 0x00007f03c805f71b in vdev_mirror_io_start (zio=0x7f0350002810) at ../../module/zfs/vdev_mirror.c:374 /openzfs#489 0x00007f03c807f243 in __zio_execute (zio=0x7f0350002810) at ../../module/zfs/zio.c:1399 /openzfs#490 zio_nowait (zio=0x7f0350002810) at ../../module/zfs/zio.c:1456 /openzfs#491 0x00007f03c8064593 in vdev_raidz_io_start (zio=0x7f0350001270) at ../../module/zfs/vdev_raidz.c:1591 /openzfs#492 0x00007f03c807f243 in __zio_execute (zio=0x7f0350001270) at ../../module/zfs/zio.c:1399 /openzfs#493 zio_nowait (zio=0x7f0350001270) at ../../module/zfs/zio.c:1456 /openzfs#494 0x00007f03c805f71b in vdev_mirror_io_start (zio=0x7f0350001e60) at ../../module/zfs/vdev_mirror.c:374 /openzfs#495 0x00007f03c807f243 in __zio_execute (zio=0x7f0350001e60) at ../../module/zfs/zio.c:1399 /openzfs#496 zio_nowait (zio=0x7f0350001e60) at ../../module/zfs/zio.c:1456 /openzfs#497 0x00007f03c805ed43 in vdev_mirror_io_done (zio=0x7f033a0c39c0) at ../../module/zfs/vdev_mirror.c:499 /openzfs#498 0x00007f03c807a0c0 in zio_vdev_io_done (zio=0x7f033a0c39c0) at ../../module/zfs/zio.c:2707 /openzfs#499 0x00007f03c808285b in __zio_execute (zio=0x7f033a0c39c0) at ../../module/zfs/zio.c:1399 /openzfs#500 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f03a8003c00, pio=0x7f033a0c39c0) at ../../module/zfs/zio.c:547 /openzfs#501 zio_done (zio=0x7f03a8003c00) at ../../module/zfs/zio.c:3278 /openzfs#502 0x00007f03c808285b in __zio_execute (zio=0x7f03a8003c00) at ../../module/zfs/zio.c:1399 /openzfs#503 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f038800c400, pio=0x7f03a8003c00) at ../../module/zfs/zio.c:547 /openzfs#504 zio_done (zio=0x7f038800c400) at ../../module/zfs/zio.c:3278 /openzfs#505 0x00007f03c808285b in __zio_execute (zio=0x7f038800c400) at ../../module/zfs/zio.c:1399 /openzfs#506 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f038800da00, pio=0x7f038800c400) at ../../module/zfs/zio.c:547 /openzfs#507 zio_done (zio=0x7f038800da00) at ../../module/zfs/zio.c:3278 /openzfs#508 0x00007f03c808285b in __zio_execute (zio=0x7f038800da00) at ../../module/zfs/zio.c:1399 /openzfs#509 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f038800fd80, pio=0x7f038800da00) at ../../module/zfs/zio.c:547 /openzfs#510 zio_done (zio=0x7f038800fd80) at ../../module/zfs/zio.c:3278 /openzfs#511 0x00007f03c807a6d3 in __zio_execute (zio=0x7f038800fd80) at ../../module/zfs/zio.c:1399 /openzfs#512 zio_execute (zio=zio@entry=0x7f038800fd80) at ../../module/zfs/zio.c:1337 /openzfs#513 0x00007f03c8060b35 in vdev_queue_io_to_issue (vq=vq@entry=0x99f8a8) at ../../module/zfs/vdev_queue.c:706 /openzfs#514 0x00007f03c806119d in vdev_queue_io_done (zio=zio@entry=0x7f03a0010950) at ../../module/zfs/vdev_queue.c:775 /openzfs#515 0x00007f03c807a0e8 in zio_vdev_io_done (zio=0x7f03a0010950) at ../../module/zfs/zio.c:2686 /openzfs#516 0x00007f03c807a6d3 in __zio_execute (zio=0x7f03a0010950) at ../../module/zfs/zio.c:1399 /openzfs#517 zio_execute (zio=0x7f03a0010950) at ../../module/zfs/zio.c:1337 /openzfs#518 0x00007f03c7fcd0c4 in taskq_thread (arg=0x966d50) at ../../lib/libzpool/taskq.c:215 /openzfs#519 0x00007f03c7fc7937 in zk_thread_helper (arg=0x967e90) at ../../lib/libzpool/kernel.c:135 /openzfs#520 0x00007f03c78890a3 in start_thread (arg=0x7f03c2703700) at pthread_create.c:309 /openzfs#521 0x00007f03c75c50fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 The backtrace was an infinite loop of `vdev_queue_io_to_issue()` invoking `zio_execute()` until it overran the stack. vdev_queue_io_to_issue() will ony invoke `zio_execute()` on raidz vdevs when aggregation I/Os are generated to improve aggregation continuity. These I/Os do not trigger any writes. However, it appears that they can be generated in such a way that they recurse infinitely upon return to `vdev_queue_io_to_issue()`. Signed-off-by: Richard Yao <[email protected]>

The below excerpt of a backtrace is from a ztest failure when running ZoL's ztest: /openzfs#453 0x00007f03c8060b35 in vdev_queue_io_to_issue (vq=vq@entry=0x99f8a8) at ../../module/zfs/vdev_queue.c:706 /openzfs#454 0x00007f03c806106e in vdev_queue_io (zio=zio@entry=0x7f0350003de0) at ../../module/zfs/vdev_queue.c:747 /openzfs#455 0x00007f03c80818c1 in zio_vdev_io_start (zio=0x7f0350003de0) at ../../module/zfs/zio.c:2659 /openzfs#456 0x00007f03c807f243 in __zio_execute (zio=0x7f0350003de0) at ../../module/zfs/zio.c:1399 /openzfs#457 zio_nowait (zio=0x7f0350003de0) at ../../module/zfs/zio.c:1456 /openzfs#458 0x00007f03c805f71b in vdev_mirror_io_start (zio=0x7f0350003a10) at ../../module/zfs/vdev_mirror.c:374 /openzfs#459 0x00007f03c807f243 in __zio_execute (zio=0x7f0350003a10) at ../../module/zfs/zio.c:1399 /openzfs#460 zio_nowait (zio=0x7f0350003a10) at ../../module/zfs/zio.c:1456 /openzfs#461 0x00007f03c806464c in vdev_raidz_io_start (zio=0x7f0350003380) at ../../module/zfs/vdev_raidz.c:1607 /openzfs#462 0x00007f03c807f243 in __zio_execute (zio=0x7f0350003380) at ../../module/zfs/zio.c:1399 /openzfs#463 zio_nowait (zio=0x7f0350003380) at ../../module/zfs/zio.c:1456 /openzfs#464 0x00007f03c805f71b in vdev_mirror_io_start (zio=0x7f0350002fb0) at ../../module/zfs/vdev_mirror.c:374 /openzfs#465 0x00007f03c807f243 in __zio_execute (zio=0x7f0350002fb0) at ../../module/zfs/zio.c:1399 /openzfs#466 zio_nowait (zio=0x7f0350002fb0) at ../../module/zfs/zio.c:1456 /openzfs#467 0x00007f03c805ed43 in vdev_mirror_io_done (zio=0x7f033957ebf0) at ../../module/zfs/vdev_mirror.c:499 /openzfs#468 0x00007f03c807a0c0 in zio_vdev_io_done (zio=0x7f033957ebf0) at ../../module/zfs/zio.c:2707 /openzfs#469 0x00007f03c808285b in __zio_execute (zio=0x7f033957ebf0) at ../../module/zfs/zio.c:1399 /openzfs#470 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f0390001330, pio=0x7f033957ebf0) at ../../module/zfs/zio.c:547 /openzfs#471 zio_done (zio=0x7f0390001330) at ../../module/zfs/zio.c:3278 /openzfs#472 0x00007f03c808285b in __zio_execute (zio=0x7f0390001330) at ../../module/zfs/zio.c:1399 /openzfs#473 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f03b4013a00, pio=0x7f0390001330) at ../../module/zfs/zio.c:547 /openzfs#474 zio_done (zio=0x7f03b4013a00) at ../../module/zfs/zio.c:3278 /openzfs#475 0x00007f03c808285b in __zio_execute (zio=0x7f03b4013a00) at ../../module/zfs/zio.c:1399 /openzfs#476 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f03b4014210, pio=0x7f03b4013a00) at ../../module/zfs/zio.c:547 /openzfs#477 zio_done (zio=0x7f03b4014210) at ../../module/zfs/zio.c:3278 /openzfs#478 0x00007f03c808285b in __zio_execute (zio=0x7f03b4014210) at ../../module/zfs/zio.c:1399 /openzfs#479 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f03b4014620, pio=0x7f03b4014210) at ../../module/zfs/zio.c:547 /openzfs#480 zio_done (zio=0x7f03b4014620) at ../../module/zfs/zio.c:3278 /openzfs#481 0x00007f03c807a6d3 in __zio_execute (zio=0x7f03b4014620) at ../../module/zfs/zio.c:1399 /openzfs#482 zio_execute (zio=zio@entry=0x7f03b4014620) at ../../module/zfs/zio.c:1337 /openzfs#483 0x00007f03c8060b35 in vdev_queue_io_to_issue (vq=vq@entry=0x99f8a8) at ../../module/zfs/vdev_queue.c:706 /openzfs#484 0x00007f03c806106e in vdev_queue_io (zio=zio@entry=0x7f0350002be0) at ../../module/zfs/vdev_queue.c:747 /openzfs#485 0x00007f03c80818c1 in zio_vdev_io_start (zio=0x7f0350002be0) at ../../module/zfs/zio.c:2659 /openzfs#486 0x00007f03c807f243 in __zio_execute (zio=0x7f0350002be0) at ../../module/zfs/zio.c:1399 /openzfs#487 zio_nowait (zio=0x7f0350002be0) at ../../module/zfs/zio.c:1456 /openzfs#488 0x00007f03c805f71b in vdev_mirror_io_start (zio=0x7f0350002810) at ../../module/zfs/vdev_mirror.c:374 /openzfs#489 0x00007f03c807f243 in __zio_execute (zio=0x7f0350002810) at ../../module/zfs/zio.c:1399 /openzfs#490 zio_nowait (zio=0x7f0350002810) at ../../module/zfs/zio.c:1456 /openzfs#491 0x00007f03c8064593 in vdev_raidz_io_start (zio=0x7f0350001270) at ../../module/zfs/vdev_raidz.c:1591 /openzfs#492 0x00007f03c807f243 in __zio_execute (zio=0x7f0350001270) at ../../module/zfs/zio.c:1399 /openzfs#493 zio_nowait (zio=0x7f0350001270) at ../../module/zfs/zio.c:1456 /openzfs#494 0x00007f03c805f71b in vdev_mirror_io_start (zio=0x7f0350001e60) at ../../module/zfs/vdev_mirror.c:374 /openzfs#495 0x00007f03c807f243 in __zio_execute (zio=0x7f0350001e60) at ../../module/zfs/zio.c:1399 /openzfs#496 zio_nowait (zio=0x7f0350001e60) at ../../module/zfs/zio.c:1456 /openzfs#497 0x00007f03c805ed43 in vdev_mirror_io_done (zio=0x7f033a0c39c0) at ../../module/zfs/vdev_mirror.c:499 /openzfs#498 0x00007f03c807a0c0 in zio_vdev_io_done (zio=0x7f033a0c39c0) at ../../module/zfs/zio.c:2707 /openzfs#499 0x00007f03c808285b in __zio_execute (zio=0x7f033a0c39c0) at ../../module/zfs/zio.c:1399 /openzfs#500 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f03a8003c00, pio=0x7f033a0c39c0) at ../../module/zfs/zio.c:547 /openzfs#501 zio_done (zio=0x7f03a8003c00) at ../../module/zfs/zio.c:3278 /openzfs#502 0x00007f03c808285b in __zio_execute (zio=0x7f03a8003c00) at ../../module/zfs/zio.c:1399 /openzfs#503 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f038800c400, pio=0x7f03a8003c00) at ../../module/zfs/zio.c:547 /openzfs#504 zio_done (zio=0x7f038800c400) at ../../module/zfs/zio.c:3278 /openzfs#505 0x00007f03c808285b in __zio_execute (zio=0x7f038800c400) at ../../module/zfs/zio.c:1399 /openzfs#506 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f038800da00, pio=0x7f038800c400) at ../../module/zfs/zio.c:547 /openzfs#507 zio_done (zio=0x7f038800da00) at ../../module/zfs/zio.c:3278 /openzfs#508 0x00007f03c808285b in __zio_execute (zio=0x7f038800da00) at ../../module/zfs/zio.c:1399 /openzfs#509 zio_notify_parent (wait=ZIO_WAIT_DONE, zio=0x7f038800fd80, pio=0x7f038800da00) at ../../module/zfs/zio.c:547 /openzfs#510 zio_done (zio=0x7f038800fd80) at ../../module/zfs/zio.c:3278 /openzfs#511 0x00007f03c807a6d3 in __zio_execute (zio=0x7f038800fd80) at ../../module/zfs/zio.c:1399 /openzfs#512 zio_execute (zio=zio@entry=0x7f038800fd80) at ../../module/zfs/zio.c:1337 /openzfs#513 0x00007f03c8060b35 in vdev_queue_io_to_issue (vq=vq@entry=0x99f8a8) at ../../module/zfs/vdev_queue.c:706 /openzfs#514 0x00007f03c806119d in vdev_queue_io_done (zio=zio@entry=0x7f03a0010950) at ../../module/zfs/vdev_queue.c:775 /openzfs#515 0x00007f03c807a0e8 in zio_vdev_io_done (zio=0x7f03a0010950) at ../../module/zfs/zio.c:2686 /openzfs#516 0x00007f03c807a6d3 in __zio_execute (zio=0x7f03a0010950) at ../../module/zfs/zio.c:1399 /openzfs#517 zio_execute (zio=0x7f03a0010950) at ../../module/zfs/zio.c:1337 /openzfs#518 0x00007f03c7fcd0c4 in taskq_thread (arg=0x966d50) at ../../lib/libzpool/taskq.c:215 /openzfs#519 0x00007f03c7fc7937 in zk_thread_helper (arg=0x967e90) at ../../lib/libzpool/kernel.c:135 /openzfs#520 0x00007f03c78890a3 in start_thread (arg=0x7f03c2703700) at pthread_create.c:309 /openzfs#521 0x00007f03c75c50fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 This occurred when ztest was simulating a scrub under heavy I/O load. Under those circumstances, it was possible for a mix of noop I/Os for aggregation continuity and the I/O elevator to generate arbitrarily deep recursion. This patch modifies ZFS to propapage a recursion counter inside the zio_t objects such that IOs will be redispatched upon reaching a given recursion depth. We can detect long call chains and dispatch to another ZIO taskq. We cut in-line when we do this to minimize the potential for taskq exhaustion that can prevent a zio from notifying its parent. Signed-off-by: Richard Yao <[email protected]>

Signed-off-by: Alex McWhirter <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes openzfs#520

…penzfs#520)

behlendorf closed this as completed in c418410 Oct 15, 2012

ryao mentioned this issue Oct 9, 2014

vdev_raidz_io_start: Ignore empty writes #2778

Closed

ryao mentioned this issue Oct 10, 2014

Redispatch ZIOs in deep call graphs #2786

Closed

behlendorf pushed a commit to behlendorf/zfs that referenced this issue May 21, 2018

_ILP32 is always defined on SPARC

466bcf3

Signed-off-by: Alex McWhirter <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes openzfs#520

pcd1193182 pushed a commit to pcd1193182/zfs that referenced this issue Sep 26, 2023

DLPX-81971 zvols and zettacache device discovery leads to deadlock (o…

1dd2749

…penzfs#520)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adjusting zfs_vdev_aggregation_limit caused lockup, corrupt pool #520

Adjusting zfs_vdev_aggregation_limit caused lockup, corrupt pool #520

akorn commented Jan 8, 2012

akorn commented Jan 8, 2012

akorn commented Jan 8, 2012

behlendorf commented Jan 13, 2012

Stoatwblr commented Jan 14, 2012

behlendorf commented Jan 17, 2012

akorn commented Jan 17, 2012

behlendorf commented Jan 17, 2012

Stoatwblr commented Jan 17, 2012

behlendorf commented Jan 18, 2012

behlendorf commented Jan 23, 2012

akorn commented Jan 23, 2012

behlendorf commented Jan 25, 2012

behlendorf commented Jun 13, 2012

Adjusting zfs_vdev_aggregation_limit caused lockup, corrupt pool #520

Adjusting zfs_vdev_aggregation_limit caused lockup, corrupt pool #520

Comments

akorn commented Jan 8, 2012

akorn commented Jan 8, 2012

akorn commented Jan 8, 2012

behlendorf commented Jan 13, 2012

Stoatwblr commented Jan 14, 2012

behlendorf commented Jan 17, 2012

akorn commented Jan 17, 2012

behlendorf commented Jan 17, 2012

Stoatwblr commented Jan 17, 2012

behlendorf commented Jan 18, 2012

behlendorf commented Jan 23, 2012

akorn commented Jan 23, 2012

behlendorf commented Jan 25, 2012

behlendorf commented Jun 13, 2012