[Protocol] Clarify handling of duplicate add/remove actions #3784
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
With more recent DBRs (14.x, 15.x) a previously written table by
delta-rs became unreadable due to the following:
24/09/26 01:12:43 ERROR Uncaught throwable from user code: com.databricks.sql.transaction.tahoe.DeltaRuntimeException: [DELTA_DUPLICATE_ACTIONS_FOUND] File operation 'remove' for path ds=2024-09-25/part-00631-d7048577-f7b0-3b87-9f2e-336d394e0387-c000.gz.parquet was specified several times.
It conflicts with ds=2024-09-25/part-00631-d7048577-f7b0-3b87-9f2e-336d394e0387-c000.gz.parquet.
It is not valid for multiple file operations with the same path to exist in a single commit.
This particular scenario resulted in a extremely rare race condition we
discovered in AWS for some delta-rs related code, but I could not find
any statements in the protocol to indicate that duplicate actions were
actually an invalid state.
I believe that they _should_ be considered an invalid state and the
validation error provided by newer Databricks runtimes to be a
reasonable one. Therefore this change adds some verbiage to the protocol
stating that DBR's behavior is acceptable for such a Delta table.
[Slack thread on the
topic](https://delta-users.slack.com/archives/C03FVMHT93Q/p1728321032555229)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With more recent DBRs (14.x, 15.x) a previously written table by delta-rs became unreadable due to the following:
This particular scenario resulted in a extremely rare race condition we discovered in AWS for some delta-rs related code, but I could not find any statements in the protocol to indicate that duplicate actions were actually an invalid state.
I believe that they should be considered an invalid state and the validation error provided by newer Databricks runtimes to be a reasonable one. Therefore this change adds some verbiage to the protocol stating that DBR's behavior is acceptable for such a Delta table.
Slack thread on the topic