mbuffer: warning: high value of number of blocks(32768): increase block size for better performance

[tomerh2001]

I am trying to perform a complete restore of my ZFS pool to my fresh server via this command:

syncoid                                 \
    -r                                  \
    --no-privilege-elevation            \
    --mbuffer-size=4G                   \
    --no-sync-snap                      \
    [email protected]:data1        \
    Pool/restore

I have free 58GB of RAM, so I could use that to speed up the process.

This is the output I'm getting from running this command:

homeserver@homeserver:/app/scripts$ sudo sh restore.sh
[sudo] password for homeserver: 
NEWEST SNAPSHOT: syncoid_homeserver_2024-08-20:22:36:48-GMT03:00
INFO: no snapshots on source newer than syncoid_homeserver_2024-08-20:22:36:48-GMT03:00 on target. Nothing to do, not syncing.
NEWEST SNAPSHOT: syncoid_homeserver_2024-08-20:22:43:44-GMT03:00
Sending incremental data1/Data@autosnap_2024-04-08_06:00:00_hourly ... syncoid_homeserver_2024-08-20:22:43:44-GMT03:00 (~ 3.2 MB):
mbuffer: warning: high value of number of blocks(32768): increase block size for better performance
mbuffer: warning: high value of number of blocks(32768): increase block size for better performance
mbuffer: warning: allocating more than half of available memory                                                  
264KiB 0:01:36 [8.22KiB/s] [=====>                                                                         ]  8% ETA 0:22:54

I get this warning: mbuffer: warning: high value of number of blocks(32768): increase block size for better performance, together with extremely slow write speeds...

So:

1. How do I increase the block size it's complaining about?
2. How do I increase the write speed... I can't restore 2TB like that...

These are some benchmarks I ran to show my hardware is perfectly capable of faster restore:

homeserver@homeserver:/app/scripts$ sudo sh benchmark.sh 
---------------------
1. Internet Speed Test
---------------------
Download Speed: 1940.50 Mbit/s
Upload Speed: 235.01 Mbit/s

---------------------
2. CPU Performance Test (FLOPS)
---------------------
CPU Performance: 6158.83 FLOPS

---------------------
3. RAM Speed Test
---------------------
RAM Write Speed: 10240.00 MiB/s
RAM Read Speed: 10240.00 MiB/s

---------------------
4. Disk Write and Read Speed Test
---------------------
Disk Write Speed: 1.8 GB/s
Disk Read Speed: 3.5 GB/s

---------------------
5. ZFS Pool Write and Read Speed Test
---------------------
ZFS Pool Write Speed: 2.8 GB/s
ZFS Pool Read Speed: 3.8 GB/s

[jimsalterjrs]

You'd have to go into the source code to change mbuffer's arguments to use larger blocks. Doing so will potentially improve total throughput, while potentially making resumes more painful if you lose connection in the middle of a sync.

More importantly, though...

mbuffer: warning: allocating more than half of available memory

If mbuffer is allocating more than half of what memory you've got left out of 58GiB, something is badly wrong and you may need to take a closer look at what you're doing with the RAM on that system.