Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-35360][SQL] RepairTableCommand respects spark.sql.addPartitionInBatch.size too #32489

Closed
wants to merge 3 commits into from

Conversation

AngersZhuuuu
Copy link
Contributor

@AngersZhuuuu AngersZhuuuu commented May 10, 2021

What changes were proposed in this pull request?

RepairTableCommand respects spark.sql.addPartitionInBatch.size too

Why are the changes needed?

Make RepairTableCommand add partition batch size configurable.

Does this PR introduce any user-facing change?

User can use spark.sql.addPartitionInBatch.size to change batch size when repair table.

How was this patch tested?

Not need

@github-actions github-actions bot added the SQL label May 10, 2021
@AngersZhuuuu
Copy link
Contributor Author

ping @MaxGekk @wangyum @maropu

Copy link
Member

@MaxGekk MaxGekk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to change the config doc since it mentions the concrete command:

"`AlterTableAddPartitionCommand` to add partitions into table. The smaller " +

@MaxGekk
Copy link
Member

MaxGekk commented May 10, 2021

RepairTableCommand respect -> RepairTableCommand respects

@AngersZhuuuu AngersZhuuuu changed the title [SPARK-35360][SQL] RepairTableCommand respect spark.sql.addPartitionInBatch.size too [SPARK-35360][SQL] RepairTableCommand respects spark.sql.addPartitionInBatch.size too May 10, 2021
@AngersZhuuuu
Copy link
Contributor Author

Need to change the config doc since it mentions the concrete command:

"`AlterTableAddPartitionCommand` to add partitions into table. The smaller " +

Done

@SparkQA
Copy link

SparkQA commented May 10, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42847/

@SparkQA
Copy link

SparkQA commented May 10, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42847/

@MaxGekk
Copy link
Member

MaxGekk commented May 10, 2021

+1, LGTM. GA passed. Merging to master.
Thank you @AngersZhuuuu .

@MaxGekk MaxGekk closed this in 7182f8c May 10, 2021
@SparkQA
Copy link

SparkQA commented May 10, 2021

Test build #138325 has finished for PR 32489 at commit 918ddcd.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants