zstd verbose output could give more info

[jimis]

Here is the output from xz -9 -vv FILE:

xz: Filter chain: --lzma2=dict=64MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
xz: 674 MiB of memory is required. The limiter is disabled.
xz: Decompression will need 65 MiB of memory.
/home/user/tmp/FILE (1/1)
  100 %     781.0 KiB / 9,367.9 KiB = 0.083                   0:02             

The extra bits that zstd is missing are, in order of importance IMO:

• Decompression memory requirements (rough number, it's not precise)
• Filter chain, i.e. the command line parameters that reproduce level 9 compression; It's super easy to copy-paste them and experiment.
• Compression memory requirements

These are printed in advance, so if the user sees unreasonable values he can abort with Ctrl-C.

[Cyan4973]

With #2847 from @Svetlitski-FB , I think this mostly answers this request.

[jimis]

Can you paste some example output?

Thank you for implementing this feature! :-)

[Cyan4973]

zstd -9 -vv FILE
*** zstd command line interface 64-bits v1.5.1, by Yann Collet ***
--zstd=wlog=14,clog=15,hlog=14,slog=5,mml=4,tlen=8,strat=6
--format=.zst --no-sparse --block-size=0 --memory=134217728 --threads=1 --content-size
Decompression will require 9954 B of memory
FILE            : 41.71%   (  9954   B =>   4152   B, FILE.zst)      .67%
FILE            : Completed in 0.00 sec  (cpu load : 79%)

[jimis]

There might be a bug. Here is me trying to compress the same pipeline. first with -19:

$ cat log.txt  | zstd -c -vv -19 > log.txt.19.zst
*** zstd command line interface 64-bits v1.5.2, by Yann Collet ***
--zstd=wlog=23,clog=24,hlog=22,slog=7,mml=3,tlen=256,strat=9
--format=.zst --block-size=0 --memory=134217728 --threads=1 --content-size
Using stdin for input 
Using stdout for output 
Sparse File Support is automatically disabled on stdout ; try --sparse 
Decompression will require 8388608 B of memory
/*stdin*\            : Completed in 7.92 sec  (cpu load : 99%)                  2.92%  
$ zstd -vv -l log.txt.19.zst 
*** zstd command line interface 64-bits v1.5.2, by Yann Collet ***
log.txt.19.zst 
# Zstandard Frames: 1
Window Size: 8388608 B (8388608 B)
Compressed Size: 619522 B (619522 B)
Check: XXH64

Note that the verbose output reports wlog=23 but afterwards it appears to be only 13 (8MB).

If I run the compression command with -19 --long I get the exact same verbose output (wlog=23) but the compression result is different.

$ cat log.txt  | zstd -c -vv -19 --long > log.txt.19long.zst
*** zstd command line interface 64-bits v1.5.2, by Yann Collet ***
--zstd=wlog=23,clog=24,hlog=22,slog=7,mml=3,tlen=256,strat=9
--format=.zst --block-size=0 --memory=134217728 --threads=1 --content-size
Using stdin for input 
Using stdout for output 
Sparse File Support is automatically disabled on stdout ; try --sparse 
Decompression will require 8388608 B of memory
/*stdin*\            : Completed in 9.18 sec  (cpu load : 99%)                  2.91%  
$ zstd -vv -l log.txt.19long.zst 
*** zstd command line interface 64-bits v1.5.2, by Yann Collet ***
log.txt.19long.zst 
# Zstandard Frames: 1
Window Size: 134217728 B (134217728 B)
Compressed Size: 616714 B (616714 B)
Check: XXH64

[Cyan4973]

Note that the verbose output reports wlog=23 but afterwards it appears to be only 13 (8MB).

23 is a correct value, it indeed means 8 MB.

So the issue is on the other side: when using -19 --long, the window size is automatically enlarged to 128 MB (by default), but this is not reflected in the verbose output report.

[jimis]

Sorry I mixed it up in the report, 13 is 8KB. :-) As you said it seems the verbose output for -19 --long is wrong and should report 27 (128MB) instead of 23.

If you want to track the issue I can submit a new ticket, just let me know.

[Cyan4973]

Sounds good

[jimis]

#3144