-
Notifications
You must be signed in to change notification settings - Fork 482
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ORC-236: Support UNION
type in Java Convert tool
#1025
Conversation
This patch add support to convert json to UNION type in orc file. For example, for schema `struct<foo:uniontype<int,string>>`, the following json lines can be parsed into UNION type. ``` {"foo": {"tag": 0, "value": 1}} {"foo": {"tag": 1, "value": "testing"}} {"foo": {"tag": 0, "value": 3}} ```
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you, @rizaon .
cc @guiyanakuang and @williamhyun , too |
UNION
type in Java Convert tool
Is this applicable to branch-1.7, @rizaon ? |
@dongjoon-hyun If backport is clean, it should be applicable. |
Thank you, @rizaon . I set the milestone, |
It might be nice to make Convert more generic, some existing json data may not have tag and value fields, adding an option for the user to customize the method of getting tag and value would be great. : ) |
Hello @guiyanakuang, thanks for your feedback!
|
@rizaon. I think it's good that this can handle simple situations. If there is complex json data we can keep upgrading |
I switched the milestone to 1.8.0 since 1.7.3 will be released in two weeks and this is a new improvement which is not a blocker for https://github.com/apache/orc/milestone/4?closed=1 . |
java/tools/src/java/org/apache/orc/tools/convert/JsonReader.java
Outdated
Show resolved
Hide resolved
+1 LGTM |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, LGTM. Thank you, @rizaon and @guiyanakuang .
Also, cc @williamhyun because he is the release manager of Apache ORC 1.7.4.
Merged to main/1.7.
### What changes were proposed in this pull request? This patch add support to convert json to UNION type in orc file. For example, for schema `struct<foo:uniontype<int,string>>`, the following json lines can be parsed into UNION type. ``` {"foo": {"tag": 0, "value": 1}} {"foo": {"tag": 1, "value": "testing"}} {"foo": {"tag": 0, "value": 3}} ``` ### Why are the changes needed? This add a missing support for UNION type in java convert tool. ### How was this patch tested? Manually test against handcrafted json file. (cherry picked from commit 6f44815) Signed-off-by: Dongjoon Hyun <[email protected]>
Thank you, @dongjoon-hyun @guiyanakuang ! |
### What changes were proposed in this pull request? This patch add support to convert json to UNION type in orc file. For example, for schema `struct<foo:uniontype<int,string>>`, the following json lines can be parsed into UNION type. ``` {"foo": {"tag": 0, "value": 1}} {"foo": {"tag": 1, "value": "testing"}} {"foo": {"tag": 0, "value": 3}} ``` ### Why are the changes needed? This add a missing support for UNION type in java convert tool. ### How was this patch tested? Manually test against handcrafted json file.
What changes were proposed in this pull request?
This patch add support to convert json to UNION type in orc file. For
example, for schema
struct<foo:uniontype<int,string>>
, the followingjson lines can be parsed into UNION type.
Why are the changes needed?
This add a missing support for UNION type in java convert tool.
How was this patch tested?
Manually test against handcrafted json file.