Skip to content

Commit

Permalink
s2: Add LZ4 block converter (#748)
Browse files Browse the repository at this point in the history
This allows converting compressed LZ4 blocks to S2 (or snappy) blocks without decompression.

LZ4 -> S2 seems to be same size on average.
LZ4 -> Snappy is usually worse.

## Single threaded performance

Speed (excluding LZ4 encoding):

```
BenchmarkLZ4Converter_ConvertBlock/html-32         	   28237	     42827 ns/op	2390.99 MB/s	       559.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/urls-32         	    2138	    541816 ns/op	1295.80 MB/s	     -3943 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/jpg-32          	  514826	      2328 ns/op	52874.24 MB/s	       482.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/jpg_200b-32     	34821668	        33.48 ns/op	5973.00 MB/s	         2.000 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/pdf-32          	  198241	      5975 ns/op	17136.81 MB/s	       136.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/html4-32        	    7002	    173440 ns/op	2361.63 MB/s	      1840 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1-32         	    5940	    196951 ns/op	 772.22 MB/s	       106.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt2-32         	    6656	    177228 ns/op	 706.32 MB/s	     -1427 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt3-32         	    2355	    510435 ns/op	 836.06 MB/s	       384.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt4-32         	    1700	    694444 ns/op	 693.88 MB/s	     -9125 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/pb-32           	   37118	     32141 ns/op	3689.60 MB/s	         1.000 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/gaviota-32      	    6961	    172253 ns/op	1070.05 MB/s	      9303 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_128b-32    	19923691	        59.82 ns/op	2139.66 MB/s	         0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_1000b-32   	 3180837	       375.2 ns/op	2665.40 MB/s	        16.00 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_10000b-32  	  184214	      6350 ns/op	1574.70 MB/s	        90.00 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_20000b-32  	   74031	     15521 ns/op	1288.54 MB/s	        -5.000 b_saved	       0 B/op	       0 allocs/op
```

Assembly speed (amd64)
```
BenchmarkLZ4Converter_ConvertBlock/html-32         	   47457	     24463 ns/op	4185.89 MB/s	       559.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/urls-32         	    3506	    330277 ns/op	2125.75 MB/s	     -3943 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/jpg-32          	  450177	      2718 ns/op	45294.89 MB/s	       482.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/jpg_200b-32     	76887589	        15.52 ns/op	12887.16 MB/s	         2.000 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/pdf-32          	  279540	      4322 ns/op	23694.21 MB/s	       136.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/html4-32        	   10000	    107485 ns/op	3810.75 MB/s	      1840 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1-32         	   10000	    117764 ns/op	1291.47 MB/s	       106.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt2-32         	   10000	    100578 ns/op	1244.60 MB/s	     -1427 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt3-32         	    3793	    313021 ns/op	1363.34 MB/s	       384.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt4-32         	    2988	    399888 ns/op	1204.99 MB/s	     -9125 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/pb-32           	   57486	     19277 ns/op	6151.76 MB/s	         1.000 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/gaviota-32      	   10000	    115641 ns/op	1593.90 MB/s	      9303 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_128b-32    	38400122	        31.21 ns/op	4101.10 MB/s	         0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_1000b-32   	 6509028	       179.9 ns/op	5559.38 MB/s	        16.00 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_10000b-32  	  368212	      3244 ns/op	3082.28 MB/s	        90.00 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_20000b-32  	  141013	      8303 ns/op	2408.69 MB/s	        -5.000 b_saved	       0 B/op	       0 allocs/op
```


Reference compression speed:

```
BenchmarkCompressBlockReference/html/default-32                    14070         82449 ns/op    1241.98 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/urls/default-32                     1215        970890 ns/op     723.14 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/jpg/default-32                    193770          5904 ns/op    20849.30 MB/s          0 B/op          0 allocs/op
BenchmarkCompressBlockReference/jpg_200b/default-32              8297767           144.1 ns/op  1387.77 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/pdf/default-32                     94203         12694 ns/op    8066.76 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/html4/default-32                   12174         97969 ns/op    4180.90 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1/default-32                     3613        333851 ns/op     455.56 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt2/default-32                     4683        260579 ns/op     480.39 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt3/default-32                     1268        947209 ns/op     450.54 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt4/default-32                     1083       1097426 ns/op     439.08 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/pb/default-32                      18357         64771 ns/op    1830.90 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/gaviota/default-32                  3942        295275 ns/op     624.23 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1_128b/default-32            11448295           105.7 ns/op  1210.45 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1_1000b/default-32            1000000          1021 ns/op     979.26 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1_10000b/default-32            116739         10114 ns/op     988.68 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1_20000b/default-32             49216         23409 ns/op     854.39 MB/s           0 B/op          0 allocs/op

BenchmarkCompressBlockReference/html/better-32                      6649        174667 ns/op     586.26 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/urls/better-32                       627       1905706 ns/op     368.41 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/jpg/better-32                      52425         22783 ns/op    5402.88 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/jpg_200b/better-32               2772865           433.3 ns/op   461.61 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/pdf/better-32                       9210        127051 ns/op     805.97 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/html4/better-32                     5835        201146 ns/op    2036.33 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1/better-32                      2034        566702 ns/op     268.38 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt2/better-32                      2386        500580 ns/op     250.07 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt3/better-32                       758       1556541 ns/op     274.17 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt4/better-32                       591       2013515 ns/op     239.31 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/pb/better-32                        7836        155117 ns/op     764.51 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/gaviota/better-32                   2473        484975 ns/op     380.06 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1_128b/better-32              4322678           275.5 ns/op   464.59 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1_1000b/better-32              468687          2533 ns/op     394.76 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1_10000b/better-32              49606         23720 ns/op     421.59 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1_20000b/better-32              14823         81300 ns/op     246.00 MB/s           0 B/op          0 allocs/op


```

Size comparisons (using Go lz4 encoder):

```
=== RUN   TestLZ4Converter_ConvertBlock/html
    lz4convert_test.go:42: input size: 102400
    lz4convert_test.go:43: lz4 size: 21195
    lz4convert_test.go:60: lz4->snappy size: 21828
    lz4convert_test.go:79: lz4->s2 size: 20636
    lz4convert_test.go:91: s2 (default) size: 20865
    lz4convert_test.go:95: s2 (better) size: 18969
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 559
    lz4convert_test.go:98: lz4 -> snappy bytes saved: -633
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 330
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 2226
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -229
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 1667
    --- PASS: TestLZ4Converter_ConvertBlock/html (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/urls
    lz4convert_test.go:42: input size: 702087
    lz4convert_test.go:43: lz4 size: 292514
    lz4convert_test.go:60: lz4->snappy size: 297926
    lz4convert_test.go:79: lz4->s2 size: 296457
    lz4convert_test.go:91: s2 (default) size: 286538
    lz4convert_test.go:95: s2 (better) size: 248076
    lz4convert_test.go:97: lz4 -> s2 bytes saved: -3943
    lz4convert_test.go:98: lz4 -> snappy bytes saved: -5412
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 5976
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 44438
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: 9919
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 48381
    --- PASS: TestLZ4Converter_ConvertBlock/urls (0.01s)
=== RUN   TestLZ4Converter_ConvertBlock/jpg
    lz4convert_test.go:42: input size: 123093
    lz4convert_test.go:43: lz4 size: 123522
    lz4convert_test.go:60: lz4->snappy size: 123040
    lz4convert_test.go:79: lz4->s2 size: 123040
    lz4convert_test.go:91: s2 (default) size: 123097
    lz4convert_test.go:95: s2 (better) size: 123097
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 482
    lz4convert_test.go:98: lz4 -> snappy bytes saved: 482
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 425
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 425
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -57
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: -57
    --- PASS: TestLZ4Converter_ConvertBlock/jpg (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/jpg_200b
    lz4convert_test.go:42: input size: 200
    lz4convert_test.go:43: lz4 size: 155
    lz4convert_test.go:60: lz4->snappy size: 153
    lz4convert_test.go:79: lz4->s2 size: 153
    lz4convert_test.go:91: s2 (default) size: 153
    lz4convert_test.go:95: s2 (better) size: 147
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 2
    lz4convert_test.go:98: lz4 -> snappy bytes saved: 2
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 2
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 8
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: 0
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 6
    --- PASS: TestLZ4Converter_ConvertBlock/jpg_200b (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/pdf
    lz4convert_test.go:42: input size: 102400
    lz4convert_test.go:43: lz4 size: 83152
    lz4convert_test.go:60: lz4->snappy size: 83428
    lz4convert_test.go:79: lz4->s2 size: 83016
    lz4convert_test.go:91: s2 (default) size: 84199
    lz4convert_test.go:95: s2 (better) size: 82884
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 136
    lz4convert_test.go:98: lz4 -> snappy bytes saved: -276
    lz4convert_test.go:99: data -> s2 (default) bytes saved: -1047
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 268
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -1183
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 132
    --- PASS: TestLZ4Converter_ConvertBlock/pdf (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/html4
    lz4convert_test.go:42: input size: 409600
    lz4convert_test.go:43: lz4 size: 81908
    lz4convert_test.go:60: lz4->snappy size: 84886
    lz4convert_test.go:79: lz4->s2 size: 80068
    lz4convert_test.go:91: s2 (default) size: 20867
    lz4convert_test.go:95: s2 (better) size: 18979
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 1840
    lz4convert_test.go:98: lz4 -> snappy bytes saved: -2978
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 61041
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 62929
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: 59201
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 61089
    --- PASS: TestLZ4Converter_ConvertBlock/html4 (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/txt1
    lz4convert_test.go:42: input size: 152089
    lz4convert_test.go:43: lz4 size: 79672
    lz4convert_test.go:60: lz4->snappy size: 79567
    lz4convert_test.go:79: lz4->s2 size: 79566
    lz4convert_test.go:91: s2 (default) size: 85931
    lz4convert_test.go:95: s2 (better) size: 71608
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 106
    lz4convert_test.go:98: lz4 -> snappy bytes saved: 105
    lz4convert_test.go:99: data -> s2 (default) bytes saved: -6259
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 8064
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -6365
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 7958
    --- PASS: TestLZ4Converter_ConvertBlock/txt1 (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/txt2
    lz4convert_test.go:42: input size: 125179
    lz4convert_test.go:43: lz4 size: 70801
    lz4convert_test.go:60: lz4->snappy size: 72231
    lz4convert_test.go:79: lz4->s2 size: 72228
    lz4convert_test.go:91: s2 (default) size: 79572
    lz4convert_test.go:95: s2 (better) size: 65938
    lz4convert_test.go:97: lz4 -> s2 bytes saved: -1427
    lz4convert_test.go:98: lz4 -> snappy bytes saved: -1430
    lz4convert_test.go:99: data -> s2 (default) bytes saved: -8771
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 4863
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -7344
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 6290
    --- PASS: TestLZ4Converter_ConvertBlock/txt2 (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/txt3
    lz4convert_test.go:42: input size: 426754
    lz4convert_test.go:43: lz4 size: 207038
    lz4convert_test.go:60: lz4->snappy size: 206693
    lz4convert_test.go:79: lz4->s2 size: 206654
    lz4convert_test.go:91: s2 (default) size: 220380
    lz4convert_test.go:95: s2 (better) size: 184936
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 384
    lz4convert_test.go:98: lz4 -> snappy bytes saved: 345
    lz4convert_test.go:99: data -> s2 (default) bytes saved: -13342
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 22102
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -13726
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 21718
    --- PASS: TestLZ4Converter_ConvertBlock/txt3 (0.01s)
=== RUN   TestLZ4Converter_ConvertBlock/txt4
    lz4convert_test.go:42: input size: 481861
    lz4convert_test.go:43: lz4 size: 277731
    lz4convert_test.go:60: lz4->snappy size: 286863
    lz4convert_test.go:79: lz4->s2 size: 286856
    lz4convert_test.go:91: s2 (default) size: 318193
    lz4convert_test.go:95: s2 (better) size: 264987
    lz4convert_test.go:97: lz4 -> s2 bytes saved: -9125
    lz4convert_test.go:98: lz4 -> snappy bytes saved: -9132
    lz4convert_test.go:99: data -> s2 (default) bytes saved: -40462
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 12744
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -31337
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 21869
    --- PASS: TestLZ4Converter_ConvertBlock/txt4 (0.01s)
=== RUN   TestLZ4Converter_ConvertBlock/pb
    lz4convert_test.go:42: input size: 118588
    lz4convert_test.go:43: lz4 size: 19003
    lz4convert_test.go:60: lz4->snappy size: 21130
    lz4convert_test.go:79: lz4->s2 size: 19002
    lz4convert_test.go:91: s2 (default) size: 18603
    lz4convert_test.go:95: s2 (better) size: 17686
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 1
    lz4convert_test.go:98: lz4 -> snappy bytes saved: -2127
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 400
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 1317
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: 399
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 1316
    --- PASS: TestLZ4Converter_ConvertBlock/pb (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/gaviota
    lz4convert_test.go:42: input size: 184320
    lz4convert_test.go:43: lz4 size: 71749
    lz4convert_test.go:60: lz4->snappy size: 63392
    lz4convert_test.go:79: lz4->s2 size: 62446
    lz4convert_test.go:91: s2 (default) size: 65016
    lz4convert_test.go:95: s2 (better) size: 55395
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 9303
    lz4convert_test.go:98: lz4 -> snappy bytes saved: 8357
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 6733
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 16354
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -2570
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 7051
    --- PASS: TestLZ4Converter_ConvertBlock/gaviota (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/txt1_128b
    lz4convert_test.go:42: input size: 128
    lz4convert_test.go:43: lz4 size: 84
    lz4convert_test.go:60: lz4->snappy size: 84
    lz4convert_test.go:79: lz4->s2 size: 84
    lz4convert_test.go:91: s2 (default) size: 80
    lz4convert_test.go:95: s2 (better) size: 76
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 0
    lz4convert_test.go:98: lz4 -> snappy bytes saved: 0
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 4
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 8
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: 4
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 8
    --- PASS: TestLZ4Converter_ConvertBlock/txt1_128b (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/txt1_1000b
    lz4convert_test.go:42: input size: 1000
    lz4convert_test.go:43: lz4 size: 807
    lz4convert_test.go:60: lz4->snappy size: 791
    lz4convert_test.go:79: lz4->s2 size: 791
    lz4convert_test.go:91: s2 (default) size: 772
    lz4convert_test.go:95: s2 (better) size: 744
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 16
    lz4convert_test.go:98: lz4 -> snappy bytes saved: 16
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 35
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 63
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: 19
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 47
    --- PASS: TestLZ4Converter_ConvertBlock/txt1_1000b (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/txt1_10000b
    lz4convert_test.go:42: input size: 10000
    lz4convert_test.go:43: lz4 size: 6969
    lz4convert_test.go:60: lz4->snappy size: 6879
    lz4convert_test.go:79: lz4->s2 size: 6879
    lz4convert_test.go:91: s2 (default) size: 6931
    lz4convert_test.go:95: s2 (better) size: 6216
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 90
    lz4convert_test.go:98: lz4 -> snappy bytes saved: 90
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 38
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 753
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -52
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 663
    --- PASS: TestLZ4Converter_ConvertBlock/txt1_10000b (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/txt1_20000b
    lz4convert_test.go:42: input size: 20000
    lz4convert_test.go:43: lz4 size: 12750
    lz4convert_test.go:60: lz4->snappy size: 12755
    lz4convert_test.go:79: lz4->s2 size: 12755
    lz4convert_test.go:91: s2 (default) size: 13513
    lz4convert_test.go:95: s2 (better) size: 11489
    lz4convert_test.go:97: lz4 -> s2 bytes saved: -5
    lz4convert_test.go:98: lz4 -> snappy bytes saved: -5
    lz4convert_test.go:99: data -> s2 (default) bytes saved: -763
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 1261
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -758
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 1266
```
  • Loading branch information
klauspost authored Feb 17, 2023
1 parent 0793ca1 commit 7e21226
Show file tree
Hide file tree
Showing 14 changed files with 9,575 additions and 7,112 deletions.
2 changes: 1 addition & 1 deletion huff0/decompress.go
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ func ReadTable(in []byte, s *Scratch) (s2 *Scratch, remain []byte, err error) {
b, err := fse.Decompress(in[:iSize], s.fse)
s.fse.Out = nil
if err != nil {
return s, nil, err
return s, nil, fmt.Errorf("fse decompress returned: %w", err)
}
if len(b) > 255 {
return s, nil, errors.New("corrupt input: output table too large")
Expand Down
28 changes: 28 additions & 0 deletions internal/lz4ref/LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
Copyright (c) 2015, Pierre Curto
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.

* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.

* Neither the name of xxHash nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Loading

0 comments on commit 7e21226

Please sign in to comment.