Writing fixed_size_list column?

[bertt]

Hi, I've got a Parquet file (see zipped attachment region_points.parquet) with a column 'geometry' of type fixed_size_list.
When opening the file in https://parquetreader.com/ the schema looks like:

Question is how can I write to such a column type?

I've tried something with:

var columns = new Column[]
{
    new Column<Double?[]>("geometry")
};

var geometryWriter = rowGroup.NextColumn().LogicalWriter<Double?[]> ();

But that gives:

region_points.zip

[adamreeve]

Hi @bertt

The fixed size list type isn't part of the Parquet specification, but is an Arrow columnar format type. The example file you've attached contains an embedded Arrow schema that provides additional typing information that can be used when reading the file into the Arrow columnar format, and it's here that the fixed size list type is specified. There's some more information on this at https://arrow.apache.org/docs/cpp/parquet.html#roundtripping-arrow-types.

ParquetSharp doesn't write Arrow format data so it's not currently possible to add the Arrow schema easily. If you really wanted you could probably figure out how to serialize an IPC schema message and include it in your Parquet file metadata though. There's code in the dotnet Apache.Arrow package that does this but it's an internal implementation detail and it looks like it's not possible to just serialize a schema by itself: https://github.com/apache/arrow/blob/6a936c4ff5007045e86f65f1a6b6c3c955ad5103/csharp/src/Apache.Arrow/Ipc/ArrowStreamWriter.cs#L637

[bertt]

Hi Adam, ok thanks good to know it's not something trivial :-) Maybe I'll do some experiments with the Apache.Arrow package.