Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-47094][SQL] SPJ : Dynamically rebalance number of buckets when they are not equal #45267

Closed
wants to merge 14 commits into from

Commits on Mar 13, 2024

  1. [SPARK-47094][SQL] SPJ : Dynamically rebalance number of buckets when…

    … they are not equal
    
      ### What changes were proposed in this pull request?
    -- Allow SPJ between 'compatible' bucket funtions
    -- Add a mechanism to define 'reducible' functions, one function whose output can be 'reduced' to another for all inputs.
    
      ### Why are the changes needed?
    -- SPJ currently applies only if the partition transform expressions on both sides are identifical.
    
      ### Does this PR introduce _any_ user-facing change?
    No
    
      ### How was this patch tested?
    Added new tests in KeyGroupedPartitioningSuite
    
      ### Was this patch authored or co-authored using generative AI tooling?
    No
    szehon-ho committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    b2f7cc0 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    cda8b26 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    eee9198 View commit details
    Browse the repository at this point in the history
  4. Add licenses

    szehon-ho committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    7bf1610 View commit details
    Browse the repository at this point in the history
  5. Fix compiler warning

    szehon-ho committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    18db83a View commit details
    Browse the repository at this point in the history
  6. Review comments

    szehon-ho committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    3fdb0d7 View commit details
    Browse the repository at this point in the history

Commits on Mar 14, 2024

  1. Try to fix doc

    szehon-ho committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    23c580f View commit details
    Browse the repository at this point in the history

Commits on Mar 19, 2024

  1. Configuration menu
    Copy the full SHA
    0c6f494 View commit details
    Browse the repository at this point in the history
  2. Fix linting

    szehon-ho committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    d2de9c3 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    d3e196c View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2024

  1. Review comments

    szehon-ho committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    1fa24a4 View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2024

  1. Configuration menu
    Copy the full SHA
    8053e58 View commit details
    Browse the repository at this point in the history

Commits on Apr 3, 2024

  1. Formatting

    szehon-ho committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    ad61b0c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    0356f9e View commit details
    Browse the repository at this point in the history