-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-27080][SQL]bug fix: mergeWithMetastoreSchema with uniform lower case comparison #24001
Conversation
(cherry picked from commit f47a765)
Looks plausible, given the map keys are definitely lower-case. This doesn't modify the names that are returned in the schema. Maybe @cloud-fan ? |
ok to test |
LGTM |
Test build #103207 has finished for PR 24001 at commit
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice job Bo, +1 for me, cc @cloud-fan, we found this during query service upgrade in Baidu.
…er case comparison When reading parquet file with merging metastore schema and file schema, we should compare field names using uniform case. In current implementation, lowercase is used but one omission. And this patch fix it. Unit test Closes #24001 from codeborui/mergeSchemaBugFix. Authored-by: CodeGod <> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit a29df5f) Signed-off-by: Wenchen Fan <[email protected]>
…er case comparison When reading parquet file with merging metastore schema and file schema, we should compare field names using uniform case. In current implementation, lowercase is used but one omission. And this patch fix it. Unit test Closes #24001 from codeborui/mergeSchemaBugFix. Authored-by: CodeGod <> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit a29df5f) Signed-off-by: Wenchen Fan <[email protected]>
thanks, merging to master/2.4/2.3 |
…er case comparison When reading parquet file with merging metastore schema and file schema, we should compare field names using uniform case. In current implementation, lowercase is used but one omission. And this patch fix it. Unit test Closes apache#24001 from codeborui/mergeSchemaBugFix. Authored-by: CodeGod <> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit a29df5f) Signed-off-by: Wenchen Fan <[email protected]>
…er case comparison When reading parquet file with merging metastore schema and file schema, we should compare field names using uniform case. In current implementation, lowercase is used but one omission. And this patch fix it. Unit test Closes apache#24001 from codeborui/mergeSchemaBugFix. Authored-by: CodeGod <> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit a29df5f) Signed-off-by: Wenchen Fan <[email protected]>
…er case comparison When reading parquet file with merging metastore schema and file schema, we should compare field names using uniform case. In current implementation, lowercase is used but one omission. And this patch fix it. Unit test Closes apache#24001 from codeborui/mergeSchemaBugFix. Authored-by: CodeGod <> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit a29df5f) Signed-off-by: Wenchen Fan <[email protected]>
What changes were proposed in this pull request?
When reading parquet file with merging metastore schema and file schema, we should compare field names using uniform case. In current implementation, lowercase is used but one omission. And this patch fix it.
How was this patch tested?
Unit test