StringArray columns returned when no pulling

[bkamins]

By default when the column contains strings and pulling is disabled the type of returned column is WeakRefStrings.StringArray{String,1}. This has two consequences:

• it surprises users (see eg type-mismatch problem when push! to data frame introduced by CSV.jl v.0.5.2 changes DataFrames.jl#1812 (comment)) - who expect Vector{String}
• it has bad memory performance consequences if you later heavily mutate the vector (you can get out-of-memory problems, as the buffer grows with each change)

There are two things I would propose:

• improve the documentation (@quinnj you probably should know what is best to write :))
• discuss if we should provide some way to get a "normal" Vector{String} in the output.

[bkamins]

Another problem with WeakRefStrings.StringArray{String,1} is that it does not support deleteat!, which is needed in filter! in the DataFrames.jl package.

[nalimilan]

Another problem with WeakRefStrings.StringArray{String,1} is that it does not support deleteat!, which is needed in filter! in the DataFrames.jl package.

AFAIK this has just been fixed by JuliaData/WeakRefStrings.jl#61.

Regarding the memory issues, maybe StringArray should compress the buffer periodically from setindex! based on some heuristics?

[bkamins]

Excellent!

[bkamins]

@quinnj Here is a list of methods that we should support if possible to be consistent with Vector API:

• deleteat!
• insert!
• splice!
• prepend!

[nalimilan]

pop! too?

[bkamins]

Right. I missed it, because it does not have Vector in the signature in the Julia documentation and I did an automatic scan.

[quinnj]

We need to make sure PooledArray/CategoricalArray support all those too, right?

[nalimilan]

Yes. See JuliaData/PooledArrays.jl#23, JuliaData/CategoricalArrays.jl#192 and JuliaLang/julia#32065. It seems that deleteat! is the most commonly requested function because DataFrames needs it for filter! and dropmissing!.

[quinnj]

I think the main issues here are all resolved now (missing mutation methods for custom arrays)