Using mainframer with rsync and local machine on Windows

[ILAgent]

Hi!
We use mainframer for building one big android project. And we use rsync for copying files from local to remote and back. The issue is sync from remote to local is too long (about 90 seconds) when local machine is on Windows. If local machine is on Mac copying takes about 5-10 seconds.
I guess it because a lot of files are copying and NTFS is slow in that case unlike Mac OS, but I'm not sure.
Did anybody have similar problems ? Any ideas ?
Thank you.

Some stats:

Number of files: 24,124 (reg: 21,586, dir: 2,538)
Number of created files: 1 (reg: 1)
Number of deleted files: 0
Number of regular files transferred: 7,300
Total file size: 260,068,359 bytes
Total transferred file size: 87,700,977 bytes
Literal data: 6,900,458 bytes
Matched data: 80,800,519 bytes
File list size: 454,298
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 518,600
Total bytes received: 5,694,927

sent 518,600 bytes received 5,694,927 bytes 70,209.34 bytes/sec
total size is 260,068,359 speedup is 41.86
Sync done: took 88 seconds.

[Pozzoooo]

Well, we don't use Windows, so I'm not sure about the specific issue.
But our project used to sync 500MB files, and the sync was slow anyway, so what we did to improve is to sync the executable only (in our case apk), ignoring everything in the build but the output folder.

[Tagakov]

Syncing the whole build folder is necessary because it contains generated code which is used by IDE.

I think zipping all build folder and transmitting it as a single file can give some boost, but I am not sure.

[artem-zinnatullin]

From our experience zipping won't really help in compare to rsync compression, but I'd definitely try 9 as compression level (you can set it for both local and remote, but remote is the main one in your case), see https://github.com/gojuno/mainframer/blob/development/docs/CONFIGURATION.md#config

Please report back with results!

[ILAgent]

@artem-zinnatullin thank you, but tried already with no success (

[Tagakov]

Changing compression level doesn't affect speed significantly. The main reason (possible) why zipping could help is moving responsibility of creating huge number of files (hai ntfs) to native zip-tool realization. I am not sure that this could actually help.

[artem-zinnatullin]

You can try scp + tar + gzip/pigz version of Mainframer and report back, you should be able to adapt old code to new in ~5 minutes :)

No changes required for remote machine, unless you want to squeeze more performance with pigz (just install pigz on local and remote machine then)

[PaulWoitaschek]

It would be great to have some kind of status report what's taking so long. Is it the compression or the network?

Even on my local network a full sync of a larger project takes

Sync remote → local machine…
Sync done: took 2 minutes 31 seconds.

[ILAgent]

It looks like a sync so slow because a lot files are syncing and antivirus is enabled (I have no permission for disable). Sync with "scp + tar + gzip/pigz" is faster a lot, but in that case unpacking takes too much time.
Thanks to everyone !

[artem-zinnatullin]

Great, thanks for checking scp + tar!

Would be super great if you could eliminate Antivirus from the measurement, it would really help other users to get a sense of how badly it affects the performance.

[artem-zinnatullin]

@PaulWoitaschek we've played with speed/duration logging in rsync, but design of its protocol doesn't let it calculate duration upfront (I know it's not directly related to your question), I also don't think we can pull statistics about compression out of it (only by hooking into its code and building custom binary I guess).

To measure impact of compression you can try to disable it by setting it to 0 in the config file and compare it with other levels of compression, you'll obviously start affecting network more but you will get a sense of which compression rate works best for you. This is what we did back in time and figured that compression of 1 to 3 usually reduces overall sync duration by couple seconds even on LAN.

It's also not clear if you're talking about Windows or not btw :)

[PaulWoitaschek]

I'm on Mac (client) to Linux (host)