Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

use row encoding for SortExec #5292

Closed
wants to merge 44 commits into from
Closed

use row encoding for SortExec #5292

wants to merge 44 commits into from

Commits on Feb 10, 2023

  1. modify sort_batch to use arrow row format for multi-column sorts

    @jaylmiller
    jaylmiller committed Feb 10, 2023
    Configuration menu
    Copy the full SHA
    ec44910 View commit details
    Browse the repository at this point in the history

Commits on Feb 11, 2023

  1. use row encoding for in memory partial sorting within SortExec

    @jaylmiller
    jaylmiller committed Feb 11, 2023
    Configuration menu
    Copy the full SHA
    c7c43e4 View commit details
    Browse the repository at this point in the history

Commits on Feb 12, 2023

  1. Revert preserving row encoding changes

    @jaylmiller
    jaylmiller committed Feb 12, 2023
    Configuration menu
    Copy the full SHA
    2e1c143 View commit details
    Browse the repository at this point in the history

Commits on Feb 13, 2023

  1. add bench for SortExec

    @jaylmiller
    jaylmiller committed Feb 13, 2023
    Configuration menu
    Copy the full SHA
    2ab03ee View commit details
    Browse the repository at this point in the history

  2. fix preserve partitioning case to run every partition instead of just… 

    … first
    @jaylmiller
    jaylmiller committed Feb 13, 2023
    Configuration menu
    Copy the full SHA
    11be061 View commit details
    Browse the repository at this point in the history

Commits on Feb 14, 2023

  1. rough draft: sorting works. 

    still need to update metrics tracking and spilling row format to disk
    @jaylmiller
    jaylmiller committed Feb 14, 2023
    Configuration menu
    Copy the full SHA
    730a89c View commit details
    Browse the repository at this point in the history

  2. Merge branch 'apache:master' into master

    @jaylmiller
    jaylmiller authored Feb 14, 2023
    Configuration menu
    Copy the full SHA
    4b1ea08 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'sort-preserve-row-encoding'

    @jaylmiller
    jaylmiller committed Feb 14, 2023
    Configuration menu
    Copy the full SHA
    a64705f View commit details
    Browse the repository at this point in the history

  4. remove some todos

    @jaylmiller
    jaylmiller committed Feb 14, 2023
    Configuration menu
    Copy the full SHA
    7aaddb5 View commit details
    Browse the repository at this point in the history

  5. fix clippy warnings

    @jaylmiller
    jaylmiller committed Feb 14, 2023
    Configuration menu
    Copy the full SHA
    96a2e15 View commit details
    Browse the repository at this point in the history

  6. checkpointing

    @jaylmiller
    jaylmiller committed Feb 14, 2023
    Configuration menu
    Copy the full SHA
    a3c632c View commit details
    Browse the repository at this point in the history

  7. SortPreservingMergeStream emits row encodings when used from SortExec

    @jaylmiller
    jaylmiller committed Feb 14, 2023
    Configuration menu
    Copy the full SHA
    c51b23c View commit details
    Browse the repository at this point in the history

Commits on Feb 15, 2023

  1. spill logic working (w/ a temporary serialization format solution)

    @jaylmiller
    jaylmiller committed Feb 15, 2023
    Configuration menu
    Copy the full SHA
    0f7bfc3 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'sort-exec'

    @jaylmiller
    jaylmiller committed Feb 15, 2023
    Configuration menu
    Copy the full SHA
    cdf72d8 View commit details
    Browse the repository at this point in the history

  3. add row encoding sizes to spill calculations. 

    update memory sizes in spill related unit tests to account for rows

limit sorts dont use row encoding.
    @jaylmiller
    jaylmiller committed Feb 15, 2023
    Configuration menu
    Copy the full SHA
    e313134 View commit details
    Browse the repository at this point in the history

  4. clean comments and small todos

    @jaylmiller
    jaylmiller committed Feb 15, 2023
    Configuration menu
    Copy the full SHA
    8b2450b View commit details
    Browse the repository at this point in the history

Commits on Feb 16, 2023

  1. cleanup SortedStream types 

    - use SortedStream instead of dynamic dispatch everywhere, gets rid of 
all the Box::pin being when passing streams between funcs
    @jaylmiller
    jaylmiller committed Feb 16, 2023
    Configuration menu
    Copy the full SHA
    085a871 View commit details
    Browse the repository at this point in the history

  2. add SortExec input case to each merge bench case 

    fix lil typo error in sort bench
    @jaylmiller
    jaylmiller committed Feb 16, 2023
    Configuration menu
    Copy the full SHA
    4196a25 View commit details
    Browse the repository at this point in the history

  3. row serialization format

    @jaylmiller
    jaylmiller committed Feb 16, 2023
    Configuration menu
    Copy the full SHA
    d4f5c10 View commit details
    Browse the repository at this point in the history

Commits on Feb 17, 2023

  1. RowBatch construction re-use row ref offsets instead of just appending

    @jaylmiller
    jaylmiller committed Feb 17, 2023
    Configuration menu
    Copy the full SHA
    33c611c View commit details
    Browse the repository at this point in the history

  2. Merge branch 'apache:main' into master

    @jaylmiller
    jaylmiller authored Feb 17, 2023
    Configuration menu
    Copy the full SHA
    331d205 View commit details
    Browse the repository at this point in the history

Commits on Feb 18, 2023

  1. Merge branch 'apache:main' into master

    @jaylmiller
    jaylmiller authored Feb 18, 2023
    Configuration menu
    Copy the full SHA
    cba6d30 View commit details
    Browse the repository at this point in the history

  2. dont need to keep array refs if we use rows

    @jaylmiller
    jaylmiller committed Feb 18, 2023
    Configuration menu
    Copy the full SHA
    1513f9a View commit details
    Browse the repository at this point in the history

  3. dont use channel for SortedSizedStream in sort (emit tuple)

    @jaylmiller
    jaylmiller committed Feb 18, 2023
    Configuration menu
    Copy the full SHA
    2ebcbc7 View commit details
    Browse the repository at this point in the history

  4. add unit test for edge case where we skip spilling the row data

    @jaylmiller
    jaylmiller committed Feb 18, 2023
    Configuration menu
    Copy the full SHA
    7354952 View commit details
    Browse the repository at this point in the history

  5. fix sort bench to actually use full data set in non-preserve partitio… 

    …n case
    @jaylmiller
    jaylmiller committed Feb 18, 2023
    Configuration menu
    Copy the full SHA
    e6fe175 View commit details
    Browse the repository at this point in the history

Commits on Feb 20, 2023

  1. Merge branch 'apache:main' into master

    @jaylmiller
    jaylmiller committed Feb 20, 2023
    Configuration menu
    Copy the full SHA
    353815b View commit details
    Browse the repository at this point in the history

  2. clippy

    @jaylmiller
    jaylmiller committed Feb 20, 2023
    Configuration menu
    Copy the full SHA
    ec49492 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'apache:main' into master

    @jaylmiller
    jaylmiller authored Feb 20, 2023
    Configuration menu
    Copy the full SHA
    60b8e6f View commit details
    Browse the repository at this point in the history

Commits on Feb 21, 2023

  1. add data skewed to first partition case for the tuple sorts in bench

    @jaylmiller
    jaylmiller committed Feb 21, 2023
    Configuration menu
    Copy the full SHA
    e430470 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'apache:main' into master

    @jaylmiller
    jaylmiller authored Feb 21, 2023
    Configuration menu
    Copy the full SHA
    a55d34e View commit details
    Browse the repository at this point in the history

  3. clippy err

    @jaylmiller
    jaylmiller committed Feb 21, 2023
    Configuration menu
    Copy the full SHA
    279c6f5 View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2023

  1. Merge branch 'apache:main' into master

    @jaylmiller
    jaylmiller authored Feb 23, 2023
    Configuration menu
    Copy the full SHA
    d42f380 View commit details
    Browse the repository at this point in the history

Commits on Mar 2, 2023

  1. Merge branch 'apache:main' into master

    @jaylmiller
    jaylmiller authored Mar 2, 2023
    Configuration menu
    Copy the full SHA
    eeb1e9c View commit details
    Browse the repository at this point in the history

Commits on Mar 3, 2023

  1. Merge branch 'apache:main' into master

    @jaylmiller
    jaylmiller authored Mar 3, 2023
    Configuration menu
    Copy the full SHA
    ba08237 View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2023

  1. Merge branch 'apache:main' into master

    @jaylmiller
    jaylmiller authored Mar 4, 2023
    Configuration menu
    Copy the full SHA
    e80578b View commit details
    Browse the repository at this point in the history

  2. use mergesort in the merge step of sort exec

    @jaylmiller
    jaylmiller committed Mar 4, 2023
    Configuration menu
    Copy the full SHA
    b82545e View commit details
    Browse the repository at this point in the history

  3. remove experimental bench cases

    @jaylmiller
    jaylmiller committed Mar 4, 2023
    Configuration menu
    Copy the full SHA
    c685a0d View commit details
    Browse the repository at this point in the history

  4. move gating logic outside of sort_batch

    @jaylmiller
    jaylmiller committed Mar 4, 2023
    Configuration menu
    Copy the full SHA
    08b3fe5 View commit details
    Browse the repository at this point in the history

  5. dont use row encoding on single batch code path

    @jaylmiller
    jaylmiller committed Mar 4, 2023
    Configuration menu
    Copy the full SHA
    790546f View commit details
    Browse the repository at this point in the history

Commits on Mar 6, 2023

  1. batch insertion order fix

    @jaylmiller
    jaylmiller committed Mar 6, 2023
    Configuration menu
    Copy the full SHA
    d13912b View commit details
    Browse the repository at this point in the history

Commits on Mar 11, 2023

  1. Merge branch 'apache:main' into master

    @jaylmiller
    jaylmiller authored Mar 11, 2023
    Configuration menu
    Copy the full SHA
    fe79275 View commit details
    Browse the repository at this point in the history

Commits on Mar 14, 2023

  1. Merge branch 'apache:main' into master

    @jaylmiller
    jaylmiller authored Mar 14, 2023
    Configuration menu
    Copy the full SHA
    0a37892 View commit details
    Browse the repository at this point in the history

Commits on Mar 17, 2023

  1. Merge branch 'apache:main' into master

    @jaylmiller
    jaylmiller authored Mar 17, 2023
    Configuration menu
    Copy the full SHA
    d202638 View commit details
    Browse the repository at this point in the history