Skip to content

Transcoding codecs for compression and decompression with LZ4

License

Notifications You must be signed in to change notification settings

JuliaIO/CodecLz4.jl

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

93 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

CodecLz4

Stable Latest Build Status CodeCov

Provides transcoding codecs for compression and decompression with LZ4. Source: LZ4 The compression algorithm is similar to the compression available through Blosc.jl, but uses the LZ4 Frame format as opposed to the standard LZ4 or LZ4_HC formats.

Codecs for the standard LZ4 and LZ4_HC formats are also provided as LZ4FastCompressor and LZ4HCCompressor. These codecs follow the LZ4 streaming examples, breaking the data into blocks and prepending each compressed block with a size. Data compressed with these codecs can be decompressed with LZ4SafeDecompressor.

Caution

LZ4FastCompressor and LZ4HCCompressor output is not compatible with lz4frame and is platform dependent.

Non-streaming functions are included via lz4_compress, lz4_hc_compress, and lz4_decompress. These should work with most other standard lz4 implementations.

Installation

Pkg.add("CodecLz4")

Usage

using CodecLz4

# Some text.
text = """
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sollicitudin
mauris non nisi consectetur, a dapibus urna pretium. Vestibulum non posuere
erat. Donec luctus a turpis eget aliquet. Cras tristique iaculis ex, eu
malesuada sem interdum sed. Vestibulum ante ipsum primis in faucibus orci luctus
et ultrices posuere cubilia Curae; Etiam volutpat, risus nec gravida ultricies,
erat ex bibendum ipsum, sed varius ipsum ipsum vitae dui.
"""

# Streaming API.
stream = LZ4FrameCompressorStream(IOBuffer(text))
for line in eachline(LZ4FrameDecompressorStream(stream))
println(line)
end
close(stream)

# Array API.
compressed = transcode(LZ4FrameCompressor, text)
@assert sizeof(compressed) < sizeof(text)
@assert transcode(LZ4FrameDecompressor, compressed) == Vector{UInt8}(text)

The API is heavily based off of CodecZLib, and uses TranscodingStreams.jl. See those for details.