You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Currently, we can end up with very-thin-strip like row groups when writing wide columns in Parquet which can hurt reader's performance quite a bit (reading metadata for each row group). I wish we could remove the default 128MB limit on writer and use 1M rows limit to end up with fairly squared wide-tables.
Describe the solution you'd like
Remove the default 128MB row group limit unless explicitly specified by the user as options.
Describe alternatives you've considered
N/A
Additional context
We also need a benchmark wide-table to measure the before and after performance.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
Currently, we can end up with very-thin-strip like row groups when writing wide columns in Parquet which can hurt reader's performance quite a bit (reading metadata for each row group). I wish we could remove the default 128MB limit on writer and use 1M rows limit to end up with fairly squared wide-tables.
Describe the solution you'd like
Remove the default 128MB row group limit unless explicitly specified by the user as options.
Describe alternatives you've considered
N/A
Additional context
We also need a benchmark wide-table to measure the before and after performance.
The text was updated successfully, but these errors were encountered: