read/write compressed csv file

[norci]

Shall we have a feature, that read/write compressed file transparently?
see:
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html

[quinnj]

You can already do:

CSV.read(GzipDecompressorStream(open(file)))

CSV.write(GzipCompressorStream(open(file)))

[xiaodaigh]

How about picking a particular file from within a zip file? Can that be done?

[quinnj]

There's nothing in CSV.jl to support processing zip archives. It looks like https://github.com/fhs/ZipFile.jl is fairly well maintained. I've avoided taking direct dependencies in CSV.jl for this kind of stuff because it composes really easily where CSV.jl can accept any IO or byte stream/array, and these other packages can provide specific compression/archive support.

[xiaodaigh]

Ok then. Perhaps an example of how to do this with zipfile.jl would be cool

[quinnj]

A PR to the docs would be very welcome! We have a whole "examples" section, so it'd be a great spot to contribute.

[xiaodaigh]

Once I figure out how to use zipfile.jl, I will. Currently, it is mystifying

[lungben]

Just for reference, this is how reading / writing of a single CSV file inside a zip file worked for me using ZipFile.jl.

Reading:

zf = ZipFile.Reader("filename.zip")
cal_df = CSV.File(read(zf.files[1])) |> DataFrame!

Writing:

filename = "myfile"
dir = ZipFile.Writer("$filename.zip")
f = ZipFile.addfile(dir, "$filename.csv", method=ZipFile.Deflate)
df |> CSV.write(f)
close(dir)

For reading, there is an example in the documentation of CSV.jl, for writing not yet.
I could make a PR if this would help.

[xiaodaigh]

I could make a PR if this would help.

From a user's perspective. It always helps