Schema Evolution Support on Delta to Iceberg

[prasanna-ds]

Hello!,

In my company, we widely use Delta tables and we want to employ Redshift as part of central infrastructure for analytics and reporting. So, we are planning to use xtable to convert Delta metadata into Iceberg metadata (tables will be part of glue data catalog). I was testing how the schema evolution works with xtable and works perfectly when there are new columns. However, if there is a column deletion or data type change, the Iceberg metadata is not getting synced (irrespective of FULL or INCREMENTAL sync).

Could you please let me know if this is the current and expected behaviour (I want to make sure I did not miss anything from the documentation)?

Thanks a lot for the help!

[the-other-tim-brown]

@prasanna-ds when you say that the metadata is not synced, do you mean you see a failure or just that the metadata json is not written and there is no failure reported?

[prasanna-ds]

@the-other-tim-brown I did some more testing.

1. Changed a datatype and wrote the delta table using overwriteSchema=True -> Glue data catalog shows old schema. There was no failure, no new/updated metadata files.
2. Added a new column to the same table on top of the above change using mergeSchema=True -> Glue data catalog still shows old schema. There was again no failure, no new/updated metadata files.

[the-other-tim-brown]

ok any logs you can share?

[prasanna-ds]

@the-other-tim-brown Sincere apologies. I overlooked the logs, Indeed there was an exception. Stacktrace below,

ERROR TableFormatSync Failed to sync table changes
java.lang.IllegalArgumentException: Cannot change column type: age_actual: double -> string
at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:463)
at org.apache.iceberg.SchemaUpdate.updateColumn(SchemaUpdate.java:287)
at org.apache.xtable.iceberg.IcebergSchemaSync.lambda$updateColumn$4(IcebergSchemaSync.java:122)
at org.apache.xtable.iceberg.IcebergSchemaSync.lambda$sync$0(IcebergSchemaSync.java:55)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at org.apache.xtable.iceberg.IcebergSchemaSync.sync(IcebergSchemaSync.java:55)
at org.apache.xtable.iceberg.IcebergConversionTarget.syncSchema(IcebergConversionTarget.java:169)
at org.apache.xtable.spi.sync.TableFormatSync.getSyncResult(TableFormatSync.java:156)
at org.apache.xtable.spi.sync.TableFormatSync.syncChanges(TableFormatSync.java:118)
at org.apache.xtable.conversion.ConversionController.syncIncrementalChanges(ConversionController.java:207)
at org.apache.xtable.conversion.ConversionController.sync(ConversionController.java:125)
at de.xxyz.redshift.xtable.handler.XTableConverterHandler.handleRequest(XTableConverterHandler.scala:87)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.amazonaws.services.lambda.runtime.api.client.EventHandlerLoader$StreamMethodRequestHandler.handleRequest(EventHandlerLoader.java:379)
at com.amazonaws.services.lambda.runtime.api.client.EventHandlerLoader$2.call(EventHandlerLoader.java:907)
at com.amazonaws.services.lambda.runtime.api.client.AWSLambda.startRuntime(AWSLambda.java:239)
at com.amazonaws.services.lambda.runtime.api.client.AWSLambda.startRuntime(AWSLambda.java:191)
at com.amazonaws.services.lambda.runtime.api.client.AWSLambda.main(AWSLambda.java:181)
ERROR ConversionController Sync failed for the following formats ICEBERG

[the-other-tim-brown]

You are limited to the schema evolutions allowed in the target format so this is expected with Iceberg