Upgrade Delta Lake version:2.4 for deletion vector support #349
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Support for Deletion Vectors has been added in Delta Lake version 2.4. The upgrade also requires updating the spark runtime version to 3.4+ In addition to chaning the version of the dependencies, this change also incorporates all the backward incompatible changes in the Delta API. 1. getSnapshotAt change: it does not accept a optional timestamp (2nd method argument) anymore. This argument was not provided / used by XTable and can safely be ignored in all invocations. 2. addFile api change: It now requires information about DeleteVector as a parameter. As Deletion vectors writing is not supported in the current version of XTable, a null is provided to the addFile method call. 3. update transaction api change: It now requires an Catalyst Expression object, instead of a generic string object, to be linked to a update operation. This change replaces the string object used by XTable with a Literal-expression. 4. getSnapshot api change: It does not require a timestamp to initialize current snapshot anymore. This change removes the additional argument in the method invocation in XTable. 5. DeltaLog metadata change: The metadata is now available through the DeltaLog's snapshot instance, instead of being made available through the DeltaLog itself like in the older versions. 6. change in the update api: it now requires the user to choose if defaults need to be ignored. It seems that the defaults need to be ingored for operations like copy. By default, the value for ignore-defaults is false for most operations. Hence it is the choosen value in XTable also. 7. Update spark version requires catalog and sql extension configurations in the sessoin definition. This change adds these two configs wherever a spark instance is created for writing Delta Lake commits.
ashvina
commented
Feb 27, 2024
| @@ -77,8 +77,6 @@ public static void initSpark() { | |||
| spark = SparkSession.builder().config(sparkConf).getOrCreate(); | |||
| } | |||
|
|
|||
| @ParameterizedTest | |||
There was a problem hiding this comment.
TODO: Need help in fixing this test. This test is currently failing with an error related to serialization.
Caused by: java.lang.AbstractMethodError: Receiver class org.apache.spark.sql.adapter.Spark3_4Adapter does not define or inherit an implementation of the resolved method 'abstract org.apache.spark.sql.avro.HoodieAvroSerializer createAvroSerializer(org.apache.spark.sql.types.DataType, org.apache.avro.Schema, boolean)' of interface org.apache.spark.sql.hudi.SparkAdapter. at org.apache.hudi.AvroConversionUtils$.createInternalRowToAvroConverter(AvroConversionUtils.scala:59)
There was a problem hiding this comment.
I'll take a look at this one but there are more tests failing.
ashvina
force-pushed
the
340-update-delta-lake-version-to-2.4+
branch
from
f11984a
to
dd72522
Compare
the-other-tim-brown
approved these changes
Feb 27, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #340
Support for Deletion Vectors has been added in Delta Lake version 2.4. The upgrade also requires updating the spark runtime version to 3.4+
In addition to chaning the version of the dependencies, this change also incorporates all the backward incompatible changes in the Delta API.