How to efficiently write array colums

[nchabra]

I am trying to write a table with a column defined as Column<float[]>.

This is some mock code with dummy data which illustrates the situation.

// This how I have it working, but it allocates a LOT.
using (var columnWriter = rowGroupWriter.NextColumn().LogicalWriter<float[]>())
{
    float[] row1 = new float[] { 1.0f, 2.0f, 3.0f, 4.0f, 5.0f };
    float[] row2 = new float[] { 1.0f, 2.0f, 3.0f, 4.0f, 5.0f };
    float[][] rows = new[] { row1, row2 };
    columnWriter.WriteBatch(rows);
}

// This how I would like to do it, but I cannot figure how.
using (var columnWriter = rowGroupWriter.NextColumn().LogicalWriter<ReadOnlyMemory<float>>())
{
    ReadOnlyMemory<float> row1 = new float[] { 1.0f, 2.0f, 3.0f, 4.0f, 5.0f };
    ReadOnlyMemory<float> row2 = new float[] { 1.0f, 2.0f, 3.0f, 4.0f, 5.0f };
    ReadOnlySpan<ReadOnlyMemory<float>> rows = new[] { row1, row2 };
    columnWriter.WriteBatch(rows);
}

Is there any hope that I can do that somehow?

[adamreeve]

Hi @nchabra, no this isn't currently possible. I think it would be a non-trivial change to support this in ParquetSharp as the use of arrays to represent Parquet logical list values is assumed throughout a lot of the code.

This would be nice to support though so we'd be open to accepting a PR implementing this.