ZIL: Improve next log block size prediction.

[amotin]

Detect single-threaded workloads by checking the previous block is fully written and flushed. It allows to make size prediction logic much more precise and skip commit delays, since we can give up on write aggregation in that case.

Since single-threaded workloads are no longer delayed, increase zfs_commit_timeout_pct from 5 to 10%. Parallel workloads should less care about it, and it should provide more aggregation.

Remove zil_min_commit_timeout tunable, since very fast ZILs should detect most of workloads as single-threaded. And when not, not delaying writes wastes extra block space allocated for aggregation.

Track history in context of bursts, not individual log blocks. It allows to not blow away all the history by single large burst of many block, and same time allows optimizations covering multiple blocks in a burst and even predicted following burst. For each burst account its optimal block size and minimal first block size. Use that statistics from the last 8 bursts to predict first block size of the next burst.

Remove predefined set of block sizes. Allocate any size we see fit, multiple of 4KB, as required by ZIL now. With compression enabled by default, ZFS already writes pretty random block sizes, so this should not surprise space allocator any more.

Allow zio_alloc_zil() to allocate bigger blocks if predicted size does not align well with pool's minimum allocation size. ZIL can make a good use of whatever block size it is given.

Reduce max_waste_space from 12 to 6% and max_copied_data from 63KB to 8KB. It allows prediction to be more precise on large bursts, improve space efficiency and reduce extra memory copying.

How Has This Been Tested?

Manual single-threaded workload tests show expected new behavior. Heavily multi-threaded workload of VMware vMotion shows improvement of ZIL space efficiency from 67 to 86%.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[amotin]

Just rebased to a new master. Would appreciate a review.

[amotin]

This needs rework on top of newer master. I'll return to it later.