-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-49205][SQL] KeyGroupedPartitioning should inherit HashPartitioningLike #47734
Conversation
cc @yaooqinn @cloud-fan @viirya @sunchao thank you |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks @ulysses-you !
Merged to master. Thank you @ulysses-you @sunchao |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, late LGTM.
BTW, KeyGroupedPartitioning
was added at Apache Spark 3.3.0 via SPARK-37377.
Do we need to backport this to branch-3.5 and branch-3.4, @ulysses-you , @sunchao , @yaooqinn ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me.
@dongjoon-hyun it is not a bug fix so the JIRA only targets 4.0.0. I'm fine to backport it to previous branch if you are are interested in. |
Thank you for a quick reply. If this is only an improvement, never mind, @ulysses-you . I was think about the original intention at Spark 3.3.0 PR~ |
…ningLike ### What changes were proposed in this pull request? This pr makes `KeyGroupedPartitioning` inherit `HashPartitioningLike`, so that the `BroadcastHashJoin#expandOutputPartitioning` and `PartitioningPreservingUnaryExecNode` can work with it. ### Why are the changes needed? To make `KeyGroupedPartitioning` support alias aware framework. ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? add test ### Was this patch authored or co-authored using generative AI tooling? no Closes apache#47734 from ulysses-you/SPARK-49205-partitioning. Authored-by: ulysses-you <[email protected]> Signed-off-by: Kent Yao <[email protected]>
…ningLike ### What changes were proposed in this pull request? This pr makes `KeyGroupedPartitioning` inherit `HashPartitioningLike`, so that the `BroadcastHashJoin#expandOutputPartitioning` and `PartitioningPreservingUnaryExecNode` can work with it. ### Why are the changes needed? To make `KeyGroupedPartitioning` support alias aware framework. ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? add test ### Was this patch authored or co-authored using generative AI tooling? no Closes apache#47734 from ulysses-you/SPARK-49205-partitioning. Authored-by: ulysses-you <[email protected]> Signed-off-by: Kent Yao <[email protected]>
What changes were proposed in this pull request?
This pr makes
KeyGroupedPartitioning
inheritHashPartitioningLike
, so that theBroadcastHashJoin#expandOutputPartitioning
andPartitioningPreservingUnaryExecNode
can work with it.Why are the changes needed?
To make
KeyGroupedPartitioning
support alias aware framework.Does this PR introduce any user-facing change?
no
How was this patch tested?
add test
Was this patch authored or co-authored using generative AI tooling?
no