Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(parquet): optionally support serde of ParquetMetaData. #4542

Closed
wants to merge 1 commit into from

Conversation

youngsofun
Copy link
Contributor

Rationale for this change

I want to use arrow-rs to read parquet files in a distributed way:
read metadata in master/coordinator, distributed them to diff slave/workers to read data.

What changes are included in this PR?

no additional dependency is added if feature serde = false (by default)
when it is true, ParquetMetaData derive(serde::Serialize, serde::Deserialize)

Are there any user-facing changes?

no

@github-actions github-actions bot added the parquet Changes to the parquet crate label Jul 19, 2023
@tustvold
Copy link
Contributor

This approach leaks a lot of internal representation information into the public API, is there a reason you can't use the thrift definitions as they have defined serialisation already?

@youngsofun
Copy link
Contributor Author

youngsofun commented Jul 20, 2023

This approach leaks a lot of internal representation information into the public API, is there a reason you can't use the thrift definitions as they have defined serialisation already?

I got your concerns, thank you. Please give me a few days to check about it.

@tustvold tustvold marked this pull request as draft July 20, 2023 14:55
@youngsofun
Copy link
Contributor Author

the thrift definitions are enough for my current usage, thank you!

@youngsofun youngsofun closed this Jul 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
parquet Changes to the parquet crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants