-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Don't return dictionary encoded booleans #42
Conversation
json_contains
in filter
I feel like what we need is something like: use arrow_array::*;
use arrow_cast::cast;
use arrow_schema::DataType;
use arrow::error::ArrowError;
/// Convert a dict array to a non-dict array.
fn unpack_dict_array(array: ArrayRef) -> Result<ArrayRef, ArrowError> {
match array.data_type() {
DataType::Dictionary(_, value_type) => {
cast(array.as_ref(), value_type)
}
_ => Ok(array)
}
} |
Co-authored-by: Samuel Colvin <[email protected]>
superceded (I think) by #43 |
) -> DataFusionResult<ArrayRef> { | ||
if return_dict { | ||
if is_json_union(values.data_type()) { | ||
// JSON union: post-process the array to set keys to null where the union member is null |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could probably move this logic into json_get
since it only applies there.
cc @davidhewitt?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks good to me!
Required by apache/datafusion#12380 and also apache/datafusion#12511.
But it's probably more efficient to operate on an array of bools than a dictionary anyway.