Optimize object creation for new Delta snapshot #326
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
CC: @vamshigv |
| deltaDataFileExtractor.iteratorWithoutStats(deltaLog.snapshot(), tableSchema)) { | ||
| fileIterator.forEachRemaining(currentDataFiles::add); | ||
| OneDataFilesDiff filesDiff = | ||
| OneDataFilesDiff.from( |
There was a problem hiding this comment.
Do we want to do something similar for Iceberg and Hudi targets? Is there any way we can make this logic common?
There was a problem hiding this comment.
That's a good point.
This logic could be applied to other clients as well. I am going to create a new issue to make this logic more general and implement it for other clients. Merging this PR in the meantime should unblock the team who is waiting for this fix. I will also mark the new issue as a good beginner issue in case any new contributor is interested.
the-other-tim-brown
approved these changes
Feb 11, 2024
ashvina
deleted the
325-optimization-avoid-object-creation-in-diff-computation
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #325
The current code in the DeltaClient generates unnecessary objects when computing the file diff to find new and removed files. The process first converts all Delta Actions of the current delta log's snapshot to OneDataFiles, uses OneDataFiles to compute the diff, and then converts the resulting OneDataFiles collection back to Delta Action objects for writing. There is a round trip from Delta Action to OneDataFiles here. For large tables with thousands of Actions in a snapshot, this results in the creation of a large number of objects unnecessarily.
This change optimizes this process by skipping the unnecessary steps of converting delta actions from the previous snapshot into OneDataFiles and then back into delta actions. This optimizations does not change the behavior of the translation.
This change is already covered by existing tests for Delta conversion