Iceberg Catalog options for metadata and glue name validation

[prasanna-ds]

Hello!
I am using XTable to convert the table format from Delta to Iceberg. While building the IcebergCatalogConfig, I pass a Catalog Option table-override.write.metadata.path to write the metadata to a different location apart from the base table. It works perfectly fine. Now, I have a scenario where we have special characters such as space, +, backticks in the table schema, and the metadata conversion fails because of glue data catalog validation error.

Caused by: java.lang.RuntimeException: SyncResult.SyncStatus(statusCode=ERROR, errorMessage=Cannot commit schema.table because Glue encountered a validation exception while accessing requested resources, errorDescription=Failed to sync FULL, canRetryOnFailure=true)

Upon some research, I learned that we can skip the validation using glue.skip-name-validation. So, I also tried passing this config in the Catalog Options and ended up with the same validation error. However, when I set only one of these configs they work perfectly fine but not together.

Could you please help me if I am missing something? or is it probably a bug?

Code (does not work):

      import org.apache.xtable.iceberg.IcebergCatalogConfig

      val icebergCatalogConfig: IcebergCatalogConfig = IcebergCatalogConfig.builder
        .catalogImpl(ICEBERG_CATALOG_IMPL)
        .catalogName("glue_database_name")
        .catalogOptions(
          Map("glue.skip-name-validation" -> "true",
            "table-override.write.metadata.path" -> "s3://iceberg-metadata/schema/table/"
          ).asJava
        )
        .build

[the-other-tim-brown]

@prasanna-ds this is likely a bug in the catalog implementation. We're just using the existing libraries and passing in the args here

[prasanna-ds]

Thanks @the-other-tim-brown for the answer. I came across your PR where there is a possibility to have a different metadata path for the target table format. If my understanding is correct, table-override.write.metadata.path option will not be needed anymore?

[the-other-tim-brown]

@prasanna-ds where is the table-override.write.metadata.path defined? I am only see a write.metadata.path option in the Iceberg codebase.

[prasanna-ds]

@the-other-tim-brown it is applied during the table creation and here.

[the-other-tim-brown]

My patch isn't accounting for the override prefix right now. Are you saying we should set both in the table metadata so it syncs with catalogs?