Is it possible to read file of more than 65536 records?

[fpenarruApr]

I have a file with featureCount = 92081 according to ogrinfo, but using ParquetSharp, it seems that I only can read 65536 records.

Any ideas?

Thanks in advance.

[marcin-krystianc]

Hi @fpenarruApr,

I don't think any such limitation exists in ParqutSharp. Can you share the code which you use to read the records, so we can try to reproduce the problem locally?

[fpenarruApr]

Hi @marcin-krystianc
Sure, please, have a look:

    using var parquetFileReader = new ParquetFileReader(file);

        using var rgReader = parquetFileReader.RowGroup(0);
        int groupNumRows = (int) rowGroupReader.MetaData.NumRows;
        var logicalReader1 = rowGroupReader.Column(0).LogicalReader<String>();

        var ids = rowGroupReader.Column(0).LogicalReader<String>().ReadAll(groupNumRows);
        Console.WriteLine("Count: " + ids.Length);

    parquetFileReader.Close();

Probably, my problem es the method ReadAll. It only accepts an int, so, I have to do a cast of NumRows.
Thanks for your help!

[marcin-krystianc]

Hi, I think the problem is that you only read rows from the first row group (RowGroup(0)). I guess there are more row groups in the file, but you are not reading rows from them.

[fpenarruApr]

Hi @marcin-krystianc

You are right! :-)

This code works:

        var parquetFileReader = new ParquetFileReader(file);
        var ids = new List<string>();
        for (var i = 0; i < parquetFileReader.FileMetaData.NumRowGroups; i++)
        {
            using (RowGroupReader rgReader = parquetFileReader.RowGroup(i))
            {
                int groupNumRows = (int) rgReader.MetaData.NumRows;
                var logicalReader1 = rgReader.Column(0).LogicalReader<String>();
                var idsChunk = rgReader.Column(0).LogicalReader<String>().ReadAll(groupNumRows);
                ids.AddRange(idsChunk);
            }
        }
        Console.WriteLine("Count: " + ids.Count);

        parquetFileReader.Close();

Thanks for your help!!

Fran