Parquet sharp generating empty file with only schema

[kaleshanagineni]

Currently we are using parquetsharp to convert data from sql query to parquet file. right now the existing behaviour is that when there is no data no parquet file generated. is there any way/workaround that we could generate parquet file with just schema/columns even when there is no data with just column defintion.

[adamreeve]

Hi @kaleshanagineni, with both the ParquetFileWriter and ParquetRowWriter classes you should get a valid parquet file created with your schema even if you don't write any data. Can you provide some example code that reproduces the problem of no file being generated?

[kaleshanagineni]

Hi @adamreeve,
thanks for getting back to me. the file do generate but its empty file with out schema(column definition). here is the snippet when I try to read the file back I'm seeing exception ParquetInvalidOrCorruptedFileException (message: 'Invalid: Parquet file size is 4 bytes, smaller than the minimum file footer (8 bytes)' .

`
var columns = new Column[]

            {

                new Column<DateTime>("Timestamp"),

                new Column<int>("ObjectId"),

                new Column<float>("Value")
               
            };

            using (var file = new ParquetFileWriter("float_timeseries.parquet", columns))
            {
                using (var rowGroup = file.AppendRowGroup())
                {

                }

            }
            using var readFile = new ParquetFileReader("float_timeseries.parquet");
            Console.WriteLine(readFile.FileMetaData.NumColumns);`

[adamreeve]

You're missing a call to file.Close(). Unfortunately just disposing of the file will close it but any errors writing be suppressed. If you add that call to file.Close() before the writer is disposed you will see an error when writing: Only -1 out of 3 columns are initialized. This error message isn't that clear, but the problem is that you've created a row group without writing any columns to it. If you instead never create a rowGroup then this works fine:

using (var file = new ParquetFileWriter("float_timeseries.parquet", columns))
{
    file.Close();
}

[adamreeve]

And if you do create a row group you need create writers for each column, even if you don't write anything to them:

using (var file = new ParquetFileWriter("/tmp/float_timeseries.parquet", columns))
{
    using (var rowGroup = file.AppendRowGroup())
    {
        var timestamps = rowGroup.NextColumn().LogicalWriter<DateTime>();
        var objectIds = rowGroup.NextColumn().LogicalWriter<int>();
        var values = rowGroup.NextColumn().LogicalWriter<float>();
    }
    file.Close();
}