CSV.Rows with CodecZlib Out-of-Memory

[cpfiffer]

I'm reading some very large gzip files, where I want an iterator for each row. I have something similar to this:

using CSV
using CodecZlib

path = "/some/giant/file.csv.gz"

open(GzipDecompressorStream, path, "r") do stream
	df = CSV.Rows(stream, header=colnames, datarow=2)
	for row in df
		# Do stuff.
	end
end

I get an OOM error for fairly large files:

reading /home/cameron/Dropbox/Research/data/oil_data_sample/WISC_TAS_20160104.csv.gz                                                                         
ERROR: LoadError: LoadError: OutOfMemoryError()
Stacktrace:
 [1] _growend! at ./array.jl:812 [inlined]
 [2] ensureroom at ./iobuffer.jl:320 [inlined]
 [3] unsafe_write(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Ptr{UInt8}, ::UInt64) at ./iobuffer.jl:409                                                       
 [4] unsafe_write at ./io.jl:509 [inlined]
 [5] macro expansion at ./gcutils.jl:87 [inlined]                                                                                                            
 [6] write at ./io.jl:532 [inlined]                                                                                                                          
 [7] write(::Base.GenericIOBuffer{Array{UInt8,1}}, ::TranscodingStreams.TranscodingStream{GzipDecompressor,IOStream}) at ./io.jl:579                         
 [8] getsource(::TranscodingStreams.TranscodingStream{GzipDecompressor,IOStream}, ::Bool) at /home/cameron/.julia/packages/CSV/9II7K/src/utils.jl:167     

Any thoughts on what do do here?

[cpfiffer]

Sorry, I hit the submit issue button too early. I've updated the main text.

[quinnj]

Thanks for the report; we can be smarter about what we're doing here.

[quinnj]

@cpfiffer, I have a PR up here; it'd be great if you could try it out for your use-case (you can get that branch by doing ] add CSV#jq/476). Hopefully, it should solve the OutOfMemory error, but as I noted on that PR, we really need a better buffering solution unique to CSV.Rows. I'll keep noodling on what we can do there.

[cpfiffer]

Seems to be working on my side with master, thanks! I'm good to close this if you are.

[quinnj]

Great!