GH-44555: [C++][Compute] Allow casting struct to bigger nullable struct #44587
+147
−66
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
|
I'm not confident that I've taken the right approach here but I think its ready for review. |
|
Ok, it looks like there is at least one more tests I need to update. I can probably sort that tomorrow. |
Tom-Newton
commented
Nov 1, 2024
| @@ -2328,7 +2328,7 @@ DatasetAndBatches MakeNestedDataset() { | |||
| field("b", boolean()), | |||
| field("c", struct_({ | |||
| field("d", int64()), | |||
| field("e", float64()), | |||
| field("e", int64()), | |||
There was a problem hiding this comment.
I suspect this was not supposed to be different from in the physical schema. This data is used to test if a virtual column will be materialised so I guess it shouldn't be testing int64 -> float64 casting at the same time.
github-actions
bot
added
awaiting committer review
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Sometimes its useful to add a column full of nulls in a cast. Currently this is supported in top level columns but not inside structs. Example where this is important: delta-io/delta-rs#1610
What changes are included in this PR?
Add support for filling in columns full of null for nullable struct fields.
I've gone for a fairly minimal change that achieves what I needed but I wonder if there should be a more significant change so that this casting is done by field name and ignore the field order.
Are these changes tested?
Yes. The expected behaviour in several existing tests has been altered and a couple of new tests have been added.
I also rolled out a custom build with this change internally because it suddenly became a critical problem.
Are there any user-facing changes?
Yes. There are scenarios that previously failed with
struct fields don't match or are in the wrong orderbut now succeed after filling innulls.