Using the to_iceberg() function I ran into a ICEBERG_COMMIT_ERROR: Failed to commit Iceberg update to the table error.

[JamieSkinner8]

Hello,
As statement above I am using the to_iceberg() function to insert a pandas dataframe into an athena iceberg table with no partitions.
This function, is being ran in parallel.
The df is one row long.

The notebook being ran in parallel works as follows:

1. Grabs a row of data from an API and outputs as a pandas df.
2. This row of delete if already in the iceberg table is deleted.
3. Use the to_iceberg() function to insert the row of data into the iceberg table.

This notebook works most the time, but sometimes fails with the message in the title. Re-running the notebook solves the issue. I am wondering if this is an issue with athena's icebergs query limits or is there something I can do with the to_iceberg function to help this?

[kukushking]

Hi @JamieSkinner8 looks like this is a known issue: apache/iceberg#1286
I do not think it's something we can handle in AWS SDK for pandas other than adding retries