-
Notifications
You must be signed in to change notification settings - Fork 159
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FEAT] UTF8 to binary coercion flag #2893
Conversation
CodSpeed Performance ReportMerging #2893 will not alter performanceComparing Summary
|
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2893 +/- ##
==========================================
+ Coverage 65.93% 66.16% +0.23%
==========================================
Files 1010 1002 -8
Lines 113440 114108 +668
==========================================
+ Hits 74794 75497 +703
+ Misses 38646 38611 -35
Flags with carried forward coverage won't be shown. Click here to find out more.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Needs tests!
pub struct ParquetSchemaInferenceOptionsBuilder { | ||
pub coerce_int96_timestamp_unit: Option<PyTimeUnit>, | ||
pub coerce_string_to_binary: Option<bool>, | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we need a builder? Could we not just implement a ParquetSchemaInferenceOptions::new()
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could, but the new
function will seem to get busy in the future... I don't mind adding another param to ParquetSchemaInferenceOptions::new
, but as we continue to add more features and flags to those options, seems like we'll end up deferring back to a builder anyways.
…r = "utf-8"` - users can specify `"raw"` instead
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Stamped with remaining comments
Overview