Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Scale table writers per task based on throughput #13111

Merged
merged 5 commits into from
Aug 30, 2022
Merged

Scale table writers per task based on throughput #13111

merged 5 commits into from
Aug 30, 2022

Commits on Aug 24, 2022

  1. Use static import for MEGABYTE in BaseHiveConnectorTest

    @gaurav8297
    gaurav8297 committed Aug 24, 2022
    Configuration menu
    Copy the full SHA
    8e9b7b8 View commit details
    Browse the repository at this point in the history

  2. Remove stream usage from getPhysicalWrittenDataSize

    @gaurav8297
    gaurav8297 committed Aug 24, 2022
    Configuration menu
    Copy the full SHA
    d8fade5 View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2022

  1. Add partition handle for exchange in PlanMatchPattern

    @gaurav8297
    gaurav8297 committed Aug 28, 2022
    Configuration menu
    Copy the full SHA
    8016abd View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2022

  1. Scale task writers adaptively based on throughput 

    Scale up local writers when current buffer memory utilization
is more than 50% of maximum and physical written
bytes by the last scaled up writer is greater than
writerMinSize. The scaling will happen upto
task.scale-writers.max-writer-count.
    @gaurav8297
    gaurav8297 committed Aug 30, 2022
    Configuration menu
    Copy the full SHA
    6ec5b36 View commit details
    Browse the repository at this point in the history

  2. Added docs for scaling task writers

    @gaurav8297
    gaurav8297 committed Aug 30, 2022
    Configuration menu
    Copy the full SHA
    b5658a6 View commit details
    Browse the repository at this point in the history