[SPARK-48260][SQL] Disable output committer coordination in one test of ParquetIOSuite #46562
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This is another approach for #46560. I am trying to avoid changing the production code for a flaky test fix here. |
viirya
reviewed
May 13, 2024
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala
Outdated
Show resolved
Hide resolved
cloud-fan
approved these changes
May 13, 2024
|
@viirya @cloud-fan thanks for the review. Merging to master. |
cloud-fan
pushed a commit
that referenced
this pull request
May 14, 2024
### What changes were proposed in this pull request? The pr aims to remove workaround for ParquetIOSuite. ### Why are the changes needed? After #46562 is completed, the reason why the ut `SPARK-7837 Do not close output writer twice when commitTask() fails` failed due to different event processing time sequence no longer exists, so we remove the previous workaround here. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? - Manually test. - Pass GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #46577 from panbingkun/SPARK-47301_FOLLOWUP. Authored-by: panbingkun <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
|
Hi, @gengliangwang and all. Do you think we can have this at release branches too? |
|
@dongjoon-hyun From the recent tests of branch-3.5(https://github.com/apache/spark/commits/branch-3.5/), the ParquetIOSuite looks fine for now. I am ok with either backporting or not backporting. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
A test from
ParquetIOSuiteis flaky:SPARK-7837 Do not close output writer twice when commitTask() failsIt turns out to be a race condition. The test injects error to the task committing step, and the job may fail in two ways:
TaskSetManagerwill abort the stage.Both these two failures are done by sending an event to
DAGScheduler, so the final job failure depends on which event gets processed first. This is not a big deal, but that test inParquetIOSuitechecks the error class. This PR fixes the flaky test by running the test case in a new test suite with output committer coordination disabledWhy are the changes needed?
fix flaky test
Does this PR introduce any user-facing change?
no
How was this patch tested?
GA test + manual test on lcoal
Was this patch authored or co-authored using generative AI tooling?
No