[SPARK-12520] [PySpark] Correct Descriptions and Add Use Cases in Equi-Join #10477
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #48327 has finished for PR 10477 at commit
|
|
@davies @JoshRosen Could you check if this PR is appropriate? Thanks! |
|
LGTM, merging into master and 1.6 (for doc). I think it's fine to not fix 1.5 branch, until some one hit it, because we may not have a new release for 1.5 branch. |
asfgit
pushed a commit
that referenced
this pull request
Dec 28, 2015
…i-Join After reading the JIRA https://issues.apache.org/jira/browse/SPARK-12520, I double checked the code. For example, users can do the Equi-Join like ```df.join(df2, 'name', 'outer').select('name', 'height').collect()``` - There exists a bug in 1.5 and 1.4. The code just ignores the third parameter (join type) users pass. However, the join type we called is `Inner`, even if the user-specified type is the other type (e.g., `Outer`). - After a PR: #8600, the 1.6 does not have such an issue, but the description has not been updated. Plan to submit another PR to fix 1.5 and issue an error message if users specify a non-inner join type when using Equi-Join. Author: gatorsmile <[email protected]> Closes #10477 from gatorsmile/pyOuterJoin.
|
Thank you! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
After reading the JIRA https://issues.apache.org/jira/browse/SPARK-12520, I double checked the code.
For example, users can do the Equi-Join like
df.join(df2, 'name', 'outer').select('name', 'height').collect()Inner, even if the user-specified type is the other type (e.g.,Outer).Plan to submit another PR to fix 1.5 and issue an error message if users specify a non-inner join type when using Equi-Join.