Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parquet writer should not write any min/max data to ColumnIndex when all values are null #6315

Closed
etseidl opened this issue Aug 27, 2024 · 2 comments · Fixed by #6316
Closed
Assignees
Labels
enhancement Any new improvement worthy of a entry in the changelog parquet Changes to the parquet crate

Comments

@etseidl
Copy link
Contributor

etseidl commented Aug 27, 2024

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
While investigating #6310 I noticed that when a page contains only null values, min and max for that page will be set to vec![0; 1]. Because of this, a byte array of length 1 will be written. Instead an empty vec should be passed so spurious min/max values are not produced.

Describe the solution you'd like
Rather than vec![0; 1], vec![] should be passed to ColumnIndexBuilder::append. This will reduce the column index size a tiny bit.

Describe alternatives you've considered
It could be the current behavior is desired, so no change would be necessary.

Additional context
The current behavior isn't a bug, but is a little wasteful.

@etseidl etseidl added the enhancement Any new improvement worthy of a entry in the changelog label Aug 27, 2024
@etseidl
Copy link
Contributor Author

etseidl commented Aug 27, 2024

take

@alamb alamb added the parquet Changes to the parquet crate label Aug 31, 2024
@alamb
Copy link
Contributor

alamb commented Aug 31, 2024

label_issue.py automatically added labels {'parquet'} from #6316

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Any new improvement worthy of a entry in the changelog parquet Changes to the parquet crate
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants