[SPARK-42252][CORE] Add spark.shuffle.localDisk.file.output.buffer and deprecate spark.shuffle.unsafe.file.output.buffer
#39819
Conversation
wayneguow
changed the title
|
cc @mccheah and @HyukjinKwon |
wayneguow
force-pushed
the
shuffle_output_buffer
branch
from
4696a36
to
bd0c1b6
Compare
Ok, it's good. I will make some changes. |
|
Gentle ping @dongjoon-hyun, the target version of this proposal has been updated. Could you go on reviewing it when you have time? Thanks. |
wayneguow
force-pushed
the
shuffle_output_buffer
branch
from
fc1b3bf
to
fe8729d
Compare
|
Thank you for your patience. Could you rebase this PR? |
docs/core-migration-guide.md
Outdated
| @@ -22,6 +22,10 @@ license: | | |||
| * Table of contents | |||
| {:toc} | |||
|
|
|||
| ## Upgrading from Core 3.4 to 3.5 | |||
|
|
|||
| - Since Spark 3.4, `spark.shuffle.unsafe.file.output.buffer` is deprecated though still works. Use `spark.shuffle.localDisk.file.output.buffer` instead. | |||
There was a problem hiding this comment.
This should be Since Spark 3.5.
There was a problem hiding this comment.
I'm here, updated it, and waiting for GA.
wayneguow
force-pushed
the
shuffle_output_buffer
branch
from
fe8729d
to
6fafbc5
Compare
dongjoon-hyun
changed the title
spark.shuffle.localDisk.file.output.buffer and deprecate spark.shuffle.unsafe.file.output.buffer
docs/core-migration-guide.md
Outdated
| @@ -22,6 +22,10 @@ license: | | |||
| * Table of contents | |||
| {:toc} | |||
|
|
|||
| ## Upgrading from Core 3.4 to 3.5 | |||
|
|
|||
| - Since Spark 3.5, `spark.shuffle.unsafe.file.output.buffer` is deprecated though still works. Use `spark.shuffle.localDisk.file.output.buffer` instead. | |||
There was a problem hiding this comment.
Remove thought still works. deprecated means it still works.
|
@wayneguow which PR made this change:
? Would be easier for me to review if you can point it out. |
@HyukjinKwon The PR is #25007 . |
There was a problem hiding this comment.
+1, LGTM from my side.
cc @HyukjinKwon , @cloud-fan , @Ngone51 , @mridulm
|
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
|
Too bad. Sorry but it seems that we forgot this PR, @wayneguow . |
|
It seems too late because Apache Spark 3.5 is RC3 stage. If we want this still, we can do for Apache Spark 4.0.0. |
wayneguow
force-pushed
the
shuffle_output_buffer
branch
from
5f19b3d
to
02843cf
Compare
|
Gentle ping @dongjoon-hyun , a pr from some time ago. |
|
friendly ping @yaooqinn Do you have time to help take a look? |
|
Merged to master, thank you @dongjoon-hyun @LuciferYang @wayneguow |
What changes were proposed in this pull request?
Deprecate spark.shuffle.unsafe.file.output.buffer and add a new config spark.shuffle.localDisk.file.output.buffer instead.
Why are the changes needed?
The old config is desgined to be used in UnsafeShuffleWriter, but now it has been used in all local shuffle writers through LocalDiskShuffleMapOutputWriter, introduced by #25007.
Does this PR introduce any user-facing change?
Old still works, advised to use new.
How was this patch tested?
Passed existing tests.