[SPARK-35239][SQL] Coalesce shuffle partition should handle empty input RDD #32362
Conversation
| // `ShuffleQueryStageExec#mapStats` returns None when the input RDD has 0 partitions, | ||
| // we should skip it when calculating the `partitionStartIndices`. | ||
| // If all input RDDs have 0 partition, we create empty partition for every shuffle reader. |
There was a problem hiding this comment.
If all input RDDs have 0 partition, the query is very fast and we don't need to optimize?
There was a problem hiding this comment.
Logically it is. But for Spark server like SparkThriftServer, we always use large shuffle partitions (e.g. 8192) and depend on aqe to coalesce it. If some users query on a empty table it will waste too many tasks. And an another issue is that driver can be busy (unnecessary task event) with the single point problem.
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
|
Test build #137993 has finished for PR 32362 at commit
|
|
Refer to this link for build results (access rights to CI server needed): |
|
thanks, merging to master! |
|
thanks for merging ! |
…ut RDD ### What changes were proposed in this pull request? Create empty partition for custom shuffle reader if input RDD is empty. ### Why are the changes needed? If input RDD partition is empty then the map output statistics will be null. And if all shuffle stage's input RDD partition is empty, we will skip it and lose the chance to coalesce partition. We can simply create a empty partition for these custom shuffle reader to reduce the partition number. ### Does this PR introduce _any_ user-facing change? Yes, the shuffle partition might be changed in AQE. ### How was this patch tested? add new test. Closes apache#32362 from ulysses-you/SPARK-35239. Authored-by: ulysses-you <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
What changes were proposed in this pull request?
Create empty partition for custom shuffle reader if input RDD is empty.
Why are the changes needed?
If input RDD partition is empty then the map output statistics will be null. And if all shuffle stage's input RDD partition is empty, we will skip it and lose the chance to coalesce partition.
We can simply create a empty partition for these custom shuffle reader to reduce the partition number.
Does this PR introduce any user-facing change?
Yes, the shuffle partition might be changed in AQE.
How was this patch tested?
add new test.