Optimal chunker parameters for Proxmox's VMA format?

[wildy]

Hello,

I'm asking for help figuring out what would be the optimal chunker parameters for dedup'ing asynchronously written Proxmox VMA images.

The VMA format is described here: https://git.proxmox.com/?p=pve-qemu.git;a=blob;f=vma_spec.txt

According to them, they use 65k cluster size with a maximum of 4M extent size, also adding a 512b header at the beginning of each extent.

Does this mean I'd need these for successful deduplication?

# My defaults (512 min, 4096k max, 64k target) as per VMA format
DEFAULT_CHUNKER='9,22,16,4095'

[RonnyPfannschmidt]

It makes no sense to tune the borg chunker which uses a hash to determine individual chunk sizes that way

Unless there was a custom chunker which is aware of the exact format boundaries its likely not to be a good match

[ThomasWaldmann]

borg 1.2 has a fixed blocksize chunker, but it is only suitable for simpler formats, like:

• block devices, LVM (just all block same size)
• simple database or other file formats with one header block and after that blocks of all same size.

So guess this is too simple for VMA and it would require a very special custom chunker, as @RonnyPfannschmidt said.

[TahmedPublic]

I think this might help you
https://github.com/n-st/borgbench
It will test different compression levels as well as different chunked settings
for a set of data and report the results.

[IngoMeyer441]

Another problem with the VMA backup is, that all data blocks can be stored out-of-order, varying between backup runs (so in a first backup blocks could be stored in the order 4,1,3,2 and in a second run they could be stored in the order 3,1,2,4). Would it even be possible to write a custom chunker to reorder the contained data blocks?

[ThomasWaldmann]

Theoretically yes, but it would be quite complex and also might be slow and need a lot of resources.

But guess one does not need to reorder blocks in the chunker IF the chunker would cut chunks at the block boundaries. If the contents of these chunks then would stay the same usually, order would not matter.