Fetching archive indexes is extremely slow - is network usage inefficient?

[callegar]

For reasons outlined in the FAQs, having a client side chunk cache may be inconvenient because it can end up taking a huge amount of space wrt the client storage abilities that may be limited. Furthermore, in many situations it only rarely occurs that the local “chunks” index needs to be rebuilt. On the other hand, in these rare occasions, not having a cache means that one spends a huge amount of time fetching archive indexes.

Now, I understand that there is a space-time trade off here and that borgception is meant to relax it one day. However, I wonder if there is something that I could try until borgception arrives. Specifically, I would like to understand if the huge amount of time needed by the index fetching may be due to some unfortunate setup in my systems and could be reduced by modifying such configuration.

What strikes me is that during this fetching:

• the client (the machine being backed up) appears to be mostly unloaded;
• the server (the machine storing the back up) appears to be mostly unloaded too;
• the network appears quite loaded, but so to say in an inefficient way: differently from what would happen with a large file transfer on the net, the network graphs do not go up and stay up, but appear to be made of quick bursts, so that the average download rate on the client stays relatively low (~ 250-500 kiB/s on a 5 GHz wifi network where this is substantially the only activity and the throughput could be much larger). At the same time, I see a non-negligible (some tens kB/s) upload rate from the client.

Is this expected? Is this the sign that there is something unfavourable on my setup? Is it the sign that the protocol involves lots of small transfers and bidirectional exchanges? Could something (e.g. in the protocol) be improved to better exploit the network throughput (e.g., by putting more CPU at use either on the client or the server or both in this procedure)?

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

Question possibly revealing an issue/inefficiency.

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

1.1.17 (both client and server)

Operating system (distribution) and version.

Ubuntu 20.04 (both client and server)

Hardware / network configuration, and filesystems used.

Client is a laptop. Server is a mini desktop. Both intel (client is way more recent and powerful, but it is my understanding that requirements on server are low). Client and server communicate via wifi (it is unfortunate that the server is temporarily on wifi too) via a wifi router/access point.
Wifi "published" connection speed is 780 Mbit/s for the client and 390 Mbit/s for the server.
Filesystem: ext 4. Client on SSD, server on rotating disk with disk on USB3 attachment.