[HUDI-2343]Fix the exception for mergeInto when the primaryKey and preCombineField of source table and target table differ in case only #3517
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dongkelun
changed the title
…eCombineField of source table and target table differ in case only
|
@pengzhiwei2018 Hi,can you please take a look? |
| sourceExpression match { | ||
| case attr: AttributeReference if attr.name.equalsIgnoreCase(targetColumnName) => true | ||
| case Cast(attr: AttributeReference, _, _) if attr.name.equalsIgnoreCase(targetColumnName) => true | ||
| case attr: AttributeReference if sourceColNameMap(attr.name.toLowerCase).equals(targetColumnName) => true |
There was a problem hiding this comment.
Can we use sparkSession.sessionState.conf.resolver to compare the column name?
There was a problem hiding this comment.
Hi,is it like this?
val resolver = sparkSession.sessionState.conf.resolver
case attr: AttributeReference if resolver(attr.name, targetColumnName) => trueI'm not sure if I understand,resolver is not case sensitive when comparing equality.However, the comparison of equality here must be case sensitive.Therefore, use sourceColNameMap(attr.name.toLowerCase) to obtain the original column name of source table without case conversion,Then compare with targetColumnName for equality.If not, add the corresponding column name with withColumn later. It is case sensitive because sourceDF is case sensitive when writing data.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the purpose of the pull request
Fix the exception for mergeInto when the primaryKey and preCombineField of source table and target table differ in case only
Verify this pull request
Unit test.