-
Notifications
You must be signed in to change notification settings - Fork 74
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changing zstd defaults (was: Update to Zstd 1.5.2) #573
Comments
zstd 1.5.2 is already available in the |
Indeed, actual benchmarks in XCP-ng are welcome 👍 This might have dual purpose:
Could you do that @sapcode ? That would be a good way to contribute! |
Running to activate xcp-ng-testing repo ;) |
@olivierlambert here are the statistics on the compare between zstd 1.4.1 vs. 1.5.2 zstd 1.4.1 Performance
zstd 1.5.2 Performance
The Winner is, best time and compression ratio zstdmt -T0 -6 --long=31 -M2308 !!!
|
Nice, however, for We could probably make the update in 8.3, what do you think @stormi ? |
I was executing the all commands on the same hardware, on the same VM which has 310GB disk size reserved - 1 x boot disk RHEL 8.3 OS 60 GB ( 12 GB used ), 1 x data disk 250 GB ( 70 GB used) - Total to be compressed 82 GB. The winner had 40 GB in 15.36 minutes. In total we had 8+ hours backup time of all VM's with the old version 1.4.1 zstd, which is now down to 3.5 hours when using zstd 1.5.2.
With regards to Zstd Parameter check this: -T#, --threads=# -6 # compression level [1-19] (default: 3) --long=31 > enables long distance matching with # windowLog, if not # is not present it defaults to 27. This increases the -M2308 > -M#, --memory=#: Set a memory usage limit. By default, Zstandard |
Thanks for the explanation on parameters, however I meant more "how those parameters could be universal for all our users"? We'll have to make some choices by default obviously :) |
IMHO it would be a good way to check how many CPU cores dom0 has and e.g. script it dynamically like: |
What about the other settings? Should we always use "long=31", the same compression level and the memory usage? If we want this upstream, we need to make it simple, however I'm not sure XAPI will accept a contribution that could be too complex. |
Hi both, |
Same as you: if we can fine better defaults than the current ones, but without causing regressions in specific situations (this is what is hard to define), then we could do the change. This would likely require a variety of benchmarks: various VM sizes, various host configurations... I doubt the exact combination of options defined above would play the role of such a default, but maybe I'm wrong. More tests may tell. My hunch is that we can define something that is possibly faster than the current defaults while remaining safe, but that the best performance is something that is defined on a case by case basis. |
As Oliver mentioned above: It should be simple and probably not too much diverting from upstream. Thus your 'per VM' idea is pretty much over the top. That custom field would add a lot of complexity, no real admin wants, as it's basically unmaintainable (RAS!). Ideally it would choose the ressources flexible, depending on dom0 properties. |
You can use only this "zstdmt -T0 -6" as default as all other values will be defaulted. T0 will select as many CPU's which are available for compression and -6 is just a higher compression then the default 3. |
Hi oliver,
the Zstd did fantastic speed upgrade in zstd release 1.5.2, is there a chance to get this version into the XCP 8.2 repo ?
https://github.com/facebook/zstd/releases
https://github.com/facebook/zstd/releases/download/v1.5.2/zstd-1.5.2.tar.gz
Or can we just replace the version in OS from here: https://rpmfind.net/linux/epel/7/x86_64/Packages/z/zstd-1.5.2-1.el7.x86_64.rpm ?
Best regards
The text was updated successfully, but these errors were encountered: